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Overview 


In 2010, the Administration for Children and Families (ACF) within the U.S. Department of 
Health and Human Services awarded the first round of five-year Health Profession Opportunity 
Grants (HPOG 1.0) to 32 organizations in 23 states; five were tribal organizations. The purpose 
of the HPOG Program is to provide education and training to Temporary Assistance for Needy 
Families (TANF) recipients and other low-income individuals for occupations in the healthcare 
field that pay well and are expected to either experience labor shortages or be in high demand. 
HPOG 1.0 grantees designed and implemented programs to provide eligible participants with 
education, occupational training, and support and employment services to help them train for 
and find jobs in a variety of healthcare professions. 


The ACF Office of Planning, Research, and Evaluation supports a multipronged research and 
evaluation strategy to assess the success of the HPOG Program. To assess its effectiveness, 
the first round of local HPOG programs was evaluated using an experimental design in which 
eligible program applicants were assigned at random to a “treatment” group that could access 
the program or a “control” group that could not. To compute the program’s impact, the outcomes 
for each group were compared. This document reports on the impacts that arose about three 
years after random assignment. It reports an overall average impact across the diverse HPOG 
1.0 programs, as well as impacts for selected subgroups of study participants. 


Research Questions 


e What impacts do the locally implemented HPOG programs as a group have on the 
outcomes of participants and their families? 


e To what extent do those impacts vary across selected subpopulations? 


e To what extent do the education and employment experiences of HPOG participants over 
time suggest that they are following a career pathway? 


Purpose 


The HPOG 1.0 Impact Study is making an important contribution to the field’s collective 
knowledge about sector-based and career pathways programs. Most other evaluations focus on 
a single program usually selected for its promise, and the results of those evaluations are 
generalizable to programs that are similar to the one evaluated. In contrast, the HPOG 1.0 
Impact Study considers a large collection of diverse, locally implemented programs, all 
Operating in their own way under broad ACF guidelines. The benefit of this approach is that it 
helps to assess whether the general HPOG model—across many implementations of it—is 
effective in achieving its goals. 


Key Findings 


According to a follow-up survey (initiated about three years after randomization) and national 
administrative data (available through the 13th follow-up quarter), relative to the control group: 


e 13 percentage points more of the HPOG treatment group had completed training (one of the 
study’s two confirmatory outcomes); and 
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quarterly earnings (the other confirmatory outcome) among the HPOG treatment group were 
no different in the 12th-13th quarters after random assignment (both groups earned 
approximately $5,000 on average per quarter). 


In addition to these two prioritized outcomes, the three-year analysis also found that, relative to 
the control group: 


1 percent more of the HPOG treatment group self-assessed as “confident” in their career 
knowledge; 


1 percentage point more of the HPOG treatment group was employed; 
12 percentage points more of the HPOG treatment group was employed in healthcare; 


2 percentage points more of the HPOG treatment group was employed at a job offering 
health insurance; 


7 percentage points more of the HPOG treatment group experienced career progress, a 
measure that combines educational progress with earnings growth; and 


3 percentage points fewer of the HPOG treatment group reported financial hardship, as 
indicated by generally not having enough money to make ends meet at the end of the month 
over the prior year. 


These findings are all statistically significantly different from zero. 


Methods 


The HPOG 1.0 Impact Study used an experimental evaluation design to assess the impacts of 
42 local HPOG programs in 23 of 32 first-round grantees nationwide. By randomizing eligible 
applicants to treatment and control groups, the evaluation provides rigorous evidence to inform 
the adult training field about sector-based and career pathways programs. The impact analysis 
used administrative data from the National Directory of New Hires (NDNH) from the first 13 
quarters after study participants were randomized and data from a follow-up survey initiated at 
about three years after they were randomized. 
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Executive Summary 


The Health Profession Opportunity Grants (HPOG) Program provides education and training to 
Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals 
for occupations in the healthcare field that pay well and are expected to either experience labor 
shortages or be in high demand.* The Administration for Children and Families (ACF) within the 
U.S. Department of Health and Human Services awarded the first round of HPOG grants 
(HPOG 1.0) to 32 grantees to run programs that met the following criteria: 


e Target skills and competencies demanded by the healthcare industry. 


e Support “career pathways”; that is, offer clearly defined routes that allow participants to build 
a career by advancing through successively higher levels of education and training, exiting 
into employment at multiple possible points. 


e Result in employer- or industry-recognized, portable education credentials (e.g., certificates 
or degrees) and professional certifications and licenses (e.g., a credential awarded by a 
Registered Apprenticeship program). 


e Combine support services with education and occupational training to help participants 
overcome barriers to succeeding in training and finding and keeping a job. 


e Provide training at times and locations that are easily accessible to targeted populations. 


ACF’s Office of Planning, Research, and Evaluation (OPRE) supports a multipronged research 
and evaluation strategy to assess the success of the HPOG Program. To assess its 
effectiveness, ACF contracted with Abt Associates to evaluate HPOG’s impact. 


Evaluation design. The evaluation used an experimental design in which eligible program 
applicants were assigned at random to a “treatment” group that could access the local HROG 
program or a “control” group that could not, and then compared their outcomes. Of 32 grantees 
awarded funding, this HPOG 1.0 Impact Study included 23 grantees that operated 42 local 
programs nationwide.” By having applicants randomized to treatment and control groups, the 
evaluation can provide strong evidence to assess the effectiveness of HPOG in pursuing its 
dual policy goals of providing training opportunities for TANF recipients and other low-income 
individuals and of providing a skilled workforce to meet the needs of the healthcare sector. 


Data sources. The study’s impact analysis used data from the National Directory of New Hires 
covering the first 13 quarters after study participants were randomized, plus data from a follow- 
up survey of study participants initiated three years after they were randomized. 


Research questions. The study’s motivating research questions are: 


1 Authority for these demonstrations is included in the Affordable Care Act (ACA), Public Law 111-148, 124 Stat. 119, 
March 23, 2010, sect. 5507(a), “Demonstration Projects to Provide Low-Income Individuals with Opportunities for 
Education, Training, and Career Advancement to Address Health Professions Workforce Needs,” adding sect. 
2008(a) to the Social Security Act, 42 U.S.C. 1397g(a). 


2 By “local program,” we mean a unique set of services, training courses, and personnel; a single grantee may 
operate one or more local programs. 
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e What impacts do the locally implemented HPOG programs as a group have on the 
outcomes of participants and their families? 


e To what extent do those impacts vary across selected subpopulations? 


e To what extent do the education and employment experiences of HPOG participants over 
time suggest that they are following a career pathway? 


Major Finding 


As of the three-year follow-up, more HPOG participants had completed occupational 
training than would have without HPOG, but this impact did not translate into greater 
earnings. These two measures—educational progress and quarterly earnings after three 
years—were pre-selected as the confirmatory outcomes for the three-year impact analysis, or 
the outcomes for which a favorable impact signals HPOG’s success in meeting its goals.? 


Key Impact Findings 


We summarize key findings for three main groups of outcomes: educational progress, labor 
market, and general well-being. In addition to these overall impacts, we also summarize some 
major subgroup impacts. The full report includes findings on additional, exploratory outcomes in 
each domain. 


Educational Progress Impacts 


e HPOG increased training completion, training duration, and participants’ self- 
assessed confidence in career knowledge and perception of progress toward long- 
term education goals. 


As shown in Exhibit ES.1, just over 75 percent of the treatment group had completed training, 
compared to nearly 63 percent of the control group, as of the three-year follow-up, representing 
roughly a 13 percentage point impact. Large shares of both the treatment and control groups 
were working (43 percent) and were already in school (26 percent) when they applied to HPOG, 
suggesting the entire sample was relatively well motivated to succeed in training. This 
motivation is evident in the levels of the outcomes for both groups. 


3 These outcomes are described as confirmatory in both the study’s Analysis Plan (Litwok et al. 2018) and the study’s 
registration page (available here: https://osf.io/Z986c/). 
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Exhibit ES.1: Educational Progress Outcomes and Impacts 
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Notes: Statistical significance levels for one-sided tests are indicated with hashtags, as follows: ### = 1 percent; ## = 5 percent; # = 10 percent. 
Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Sources: Three-year survey. 


Although the average duration of training was relatively short, HPOG increased the duration of 
training as reported through the three-year survey: the treatment group’s average time in 
training was about 742 months, about one month longer than those in the control group. There is 
wide variation across participants in the length of trainings pursued and completed. 


As shown in Exhibit ES.1, HPOG also had impacts on some less tangible aspects of 
educational progress. The treatment group’s self-assessed confidence in their career 
knowledge was slightly higher than that of the control group’s; their perception of their progress 
toward long-term education goals was higher as well. 


Labor Market Impacts 


e HPOG had no detectable impacts on quarterly earnings in the 12th-13th quarters after 
random assignment, but had impacts on employment rates, employment in 
healthcare, job quality, and career progress. 


Administrative earnings data show that both the treatment and control groups earned an 
average of approximately $5,000 over the 12th-13th quarters after random assignment, as 
shown in Exhibit ES.2. In that same time frame, about 82 percent of the treatment group was 
employed, an improvement of 1 percentage point relative to the control group. 


HPOG increased healthcare employment and improved employment conditions. According to 
the three-year survey, 56 percent of the treatment group was employed in healthcare, compared 
to 44 percent of the control group. The treatment group was also more likely to work at a job 
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that offered health insurance. Finally, HPOG increased career progress, which we define as a 
combination of training completion and earnings growth. Three years after random assignment 
43 percent of the treatment group had made career progress compared to 36 percent of the 
control group. 


Exhibit ES.2: Labor Market Outcomes and Impacts 
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Notes: Career progress reflects a combination of training completion and earnings growth (see Chapter 4 for details). 

Statistical significance levels for one-sided tests are indicated with hashtags, as follows: *## = 1 percent; ** = 5 percent; * = 10 percent. 
Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Sources: Three-year survey; National Directory of New Hires. 


Well-being Impacts 


e HPOG did not decrease individual public assistance use. However, HPOG reduced 
self-reported financial hardship and perceived stress. 


Exhibit ES.3 shows that HPOG did not decrease individual receipt of TANF, SNAP, or Medicaid 
in the prior month. HPOG also had no impact on the treatment group’s personal or household 
income. That said, HPOG reduced self-reported financial hardship: about one quarter of the 
treatment group reported that in the prior year they “generally did not have enough money to 
make ends meet at the end of the month,” which was about 3 percentage points lower than in 
the control group. The treatment group also reported a slightly lower stress level. 
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Exhibit ES.3: General Well-Being Outcomes and Impacts 
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Major Subgroup Impacts 


e HPOG increased educational progress across all major subgroups analyzed. 
Earnings gains accrued only to those who came into the program with some 
postsecondary education or more. 


This study is large enough to potentially provide insights into impacts on a wide range of 
subgroups, which we defined by demographics, certain “policy-relevant” characteristics, and 
baseline receipt of public assistance. These analyses provide evidence that educational 
progress gains are pervasive: all of the subgroups analyzed experienced educational progress 
impacts, and some had relatively greater impacts. For example, among the subgroups defined 
by certain policy-relevant traits, those that one might associate with their being less prepared for 
a program such as HPOG—not enrolled in training at baseline, with a high school education or 
less, with one or more barriers to school or work, and not employed—showed improvements in 
their educational progress that were larger than their more prepared counterparts. In addition, 
those who received SNAP or WIC at baseline experienced larger educational progress gains 
than those who were not receiving assistance at baseline. 


In comparison, most subgroups’ earnings impacts were no different from the overall study 
sample’s earnings impacts, with one exception: those who entered with at least some 
postsecondary education experienced earnings gains that were larger than those who entered 
with less than a high school degree. 


The study also examined impacts on parents and their dependent children. Parents experienced 
relatively greater educational progress gains than non-parents (though both groups improved). 
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Beyond that, parents experienced impacts similar to the full sample, and there were no 
detectable impacts—either favorable or unfavorable—on their children’s development or well- 
being. This Executive Summary mentions subgroup results only for educational progress and 
earnings. All subgroup results—including on many more outcomes—are detailed in Chapters 6 
and 7 and Appendices F and G. 


Additional Descriptive Findings 


Beyond the analysis of three-year impacts, some descriptive analyses may have implications for 
future research or program practice. 


e A large share of HPOG participants (84 percent) participated in short-term trainings, 
and those trainings associate with relatively low-paying jobs. The remaining share (16 
percent) participated in longer-term trainings that associated with relatively high- 
paying jobs. 


The most common of the short-term trainings were for Nursing Aides, Orderlies, and 
Attendants—occupations with limited earnings potential. Long-term trainings were for 
Registered Nurse, Licensed Practical Nurse, or Licensed Vocational Nurse. To improve future 
earnings, people might need to return to complete additional trainings in order to move onto the 
next—and better-paying—rung of healthcare occupations, as the career pathways framework 
intends. 


e Less than 10 percent of HPOG treatment group respondents returned to complete a 
second training, whether they first enrolled in a short-term or long-term training. 


Analysis of the patterns of employment versus training considered the extent to which HROG 
participants moved from training to employment to a second training. Very few treatment group 
members returned for a second training within this follow-up period. This pattern—which 
represents assumptions of the career pathways framework—may be so infrequently observed 
either for programmatic reasons (the local HPOG programs simply did not engage much of the 
sample into subsequent trainings) or because not enough time has passed to observe people 
returning to training. Future research will explore this further. 


Conclusion and Implications 


The HPOG 1.0 Impact Study is making an important contribution to the field’s collective 
knowledge about sector-based and career pathways programs. Most other notable evaluations 
consider one or a handful of programs. These are usually programs that have shown promise, 
and the results of those evaluations are only generalizable to other programs that are similar to 
the one evaluated. In contrast, the HPOG 1.0 Impact Study considers the average impact over a 
large collection of diverse programs, all operating in their own way under broad ACF guidelines. 
This means that the evaluation assesses whether the general HPOG model—across many 
implementations of it—is effective. 


As of this three-year follow-up, HPOG Program impacts on educational progress have not 
resulted in earnings impacts. Longer-term earnings gains seem dependent on whether those 
currently in low-paying healthcare jobs would return for additional training. If they do, then they 
may be able to advance along a healthcare career ladder to better jobs. If the treatment group 
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does not return to training and remains in the largely low-paying healthcare jobs that they are in 
three years after random assignment, then future earnings gains are less likely. The planned 
six-year follow-up should be in a position to determine which of these possibilities reflects the 
HPOG experience. 
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ACF’s HPOG Research and Evaluation Portfolio 


ACF’s Office of Planning, Research, and Evaluation (OPRE) is using a multipronged research and evaluation strategy to 
assess the implementation, outcomes, and impacts of the Health Profession Opportunity Grants (HPOG) Program. The 
program supports demonstration projects that provide Temporary Assistance for Needy Families (TANF) recipients and other 
low-income individuals with the opportunity to obtain education and training for occupations in the healthcare field that pay 
well and are expected to either experience labor shortages or be in high demand. 


HPOG First Round (HPOG 1.0) 
In 2010, ACF awarded the first round of five-year HPOG grants (HPOG 1.0) to 32 organizations in 23 states; five were tribal 
organizations. Non-tribal grantee programs served 36,624 participants over the five years. 


HPOG Implementation and Outcomes Research 
National Implementation Evaluation (NIE). The research team conducted implementation and outcomes research for the 27 
non-tribal grantees operating 49 programs in 19 states. The evaluation explored how the HPOG 1.0 programs are 
implemented across grantees (Descriptive Implementation Study), what individual-level outcomes and outputs occur 
(Outcome Study), and how HPOG influences service delivery systems (Systems Change Analysis). 


e January 2018 final report: https:/Avww.acf.hhs.gov/opre/resource/final-report-national-implementation-evaluation-of-the- 
first-round-health-profession-opportunity-grants-hpog-10 


HPOG 1.0 Impact Study 
Of the 27 first-round non-tribal grantees, 23 participated in an experimental study to assess impacts of HPOG 1.0 (the 
remaining four grantees participated in other ACF-funded research). The 23 study grantees implemented 42 programs in 92 
locations across 19 states. Beginning about 2'/ years after grants are awarded, local programs randomly assigned eligible 
applicants to a “treatment” group that could access HPOG or a “control” group that could not. The study randomized 13,802 
participants. Among treatment group members, some were offered the standard local HPOG program and some an 
“enhanced” version where some study participants were offered access to emergency assistance, non-cash incentives, or 
facilitated peer support groups. All study participants completed a baseline survey at enrollment. 


Short-Term Impacts. About 15-18 months after being randomly assigned, all study participants were invited to complete a 
follow-up survey. The research team assessed short-term outcomes using survey data, administrative data on employment 
and earnings, implementation research results for the 23 grantees from NIE, and site visits conducted specifically for the 
Impact Study. 


e May 2018 HPOG 1.0 Impact Study Interim Report: Program Implementation and Short-Term Impacts (updated as of 
November 2019): https:/Awww.acf.hhs. gov/opre/resource/health-profession-opportunity-grants-hpog-10-impact-study- 


interim-report-implementation-short-term-impacts 


Longer Term Impacts. ACF is supporting additional follow-up for HPOG 1.0 in two longer-term studies approximately three 
years and six years after random assignment. 


e Three years (this report): https:/Awww.acf.hhs. gov/opre/research/project/career-pathways-intermediate-outcomes-cpio- 
study 


e Six years: https://www.acf.hhs.gov/opre/research/project/career-pathways-long-term-outcomes-study 


For More Information 
See Appendix A for more information on HPOG research and how it is positioned in ACF’s career pathways research 
portfolio. 


OPRE’s HPOG Research and Evaluation portfolio page: https://www.acf.hhs.gov/opre/research/project/evaluation-portfolio- 
for-the-health-profession-opportunity-grants-hpog 


Career Pathways website: http://www.career-pathways.org/ 
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Terminology Used in This Report 


Terms Related to HPOG 
Funding Opportunity Announcement (FOA): the government-issued call for applications that outlined terms for funded 
grantees and programmatic expectations for their programs. 


HPOG Program: a term used to refer to the national Health Profession Opportunity Grants initiative, including all grantees 
and programs. 


HPOG grantee: an entity receiving HPOG funding that is responsible for funding and overseeing one or more local 
programs.Local HPOG program: a unique set of services, training courses, and personnel. Many grantees fund and operate 
one program; some fund multiple programs. 


Terms Related to the HPOG Evaluation 
HPOG Impact Study: an experimental study designed to assess the impact of the HPOG Program on participants across 23 
grantees. 


e HPOG-only grantees: 20 of the 32 HPOG grantees that were not otherwise participating in a study funded under ACF’s 
HPOG Research and Evaluation Portfolio. We refer to programs within these grantees as HPOG-only programs. 


e HPOGIPACE grantees: three of the 32 HPOG grantees that were also evaluated as part of ACF’s Pathways for 
Advancing Careers and Education (PACE) study. We refer to programs within these grantees as HPOG/PACE 
programs. Three additional programs are funded by HPOG-only grantees but are evaluated as part of the PACE study. 
We also refer to these as HPOG/PACE programs. 


HPOG logic model: a conceptual framework for describing how program inputs and activities are associated with anticipated 
education, training, and labor market outcomes. 


Study participants: the label for all prospective HPOG participants who enrolled in the study, regardless of the experimental 
group (treatment or control) to which they were randomly assigned. 


e Treatment group members: The study participants who were offered access to the HPOG Program as part of study 
participation. 


e Control group members: The study participants who were not offered access to the HPOG Program as part of study 
participation. 


Outcomes: The specific measures of interest that the HPOG Program aims to influence. The Impact Study defined three 
types of outcomes: 


e Confirmatory outcomes: main indicators of the extent to which the program is making progress toward its goals. 
e Secondary outcomes: additional important outcomes identified in the HPOG logic model. 


e Exploratory outcomes: additional outcomes of interest, including those that are (1) affected by the program but not 
identified in the logic model; and (2) alternative specifications of confirmatory and secondary outcomes. 


Outcome domain: A category of related outcomes. 
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Terminology Used in This Report (continued) 


Terms Related to Data Collection 
Baseline data: refers to information collected from participants at study enrollment. 


Performance Reporting System (PRS): the management information system used by all HPOG grantees to document 
program activities and accomplishments of individual program participants and to track program results against program 
goals. Data collected in the PRS was used to construct baseline measures for participants at HPOG-only grantees. 


Basic Information Form (BIF): the baseline data collection instruments used by the HPOG/PACE grantees. BIF data were 
used to construct baseline measures for participants at HPOG/PACE grantees. 


National Directory of New Hires (NDNH): a national database of quarterly earnings data, collected from states and the 
federal government. NDNH is used to construct key earnings and employment outcomes. 


Three-year survey: a survey of HPOG participants conducted approximately three years after study enrollment. Data from 
the three-year survey were used to construct key educational progress and well-being outcomes. 


General Terms 
Temporary Assistance for Needy Families (TANF): a federal program designed to help families achieve self-sufficiency by 
providing cash assistance and related supports. 


Supplemental Nutrition Assistance Program (SNAP): a federal assistance program that provides nutrition benefits to 
supplement needy families’ food budgets so they can purchase healthy food and move towards self-sufficiency. 


Special Supplemental Nutrition Program for Women, Infants, and Children (WIC): a federal assistance program that 
provides food assistance, healthcare referrals, and nutrition education to low-income pregnant women, new mothers, and 
infants and children up to age five at nutritional risk. 


Medicaid: a federal and state program that provides health insurance to low-income people, families and children, pregnant 
women, the elderly, and people with disabilities. 


Full-time equivalent (FTE) months of enrollment: a measure that summarizes the number of months an individual was 
enrolled in education or training, based on reported enrollment in the three-year survey. One month of full-time enrollment is 
equal to 1 FTE month; one month of part-time enrollment is equal to 0.5 FTE months. 
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1. Introduction 


The Health Profession Opportunity Grants (HPOG) Program responds to both the challenges 
that low-income, low-skilled individuals have in securing family-supporting jobs and the 
increasing demand for qualified healthcare professionals. 


In September 2010, the Administration for Children and Families (ACF) within the 

U.S. Department of Health and Human Services (HHS) awarded a first round of HPOG grants 
(HPOG 1.0) to 32 grantees in 23 states. Grants were awarded to organizations to provide 
education and training to Temporary Assistance for Needy Families (TANF) recipients and other 
low-income individuals for occupations in the healthcare field that pay well and are expected 
either to experience labor shortages or to be in high demand.* Grantees included government 
agencies, community-based organizations, postsecondary educational institutions, and five 
tribal-affiliated organizations.° 


To assess the effectiveness of the HPOG Program among the non-tribal grantees, ACF 
contracted with Abt Associates to conduct the HPOG 1.0 Impact Study. In a May 2018 report 
(Peck et al. 2018), the Impact Study analyzed participants’ outcomes measured about 15-18 
months after their enrollment in HPOG. This Three-Year Impacts Report describes the 
HPOG treatment group members’ experiences and impacts three years after study 
enrollment. A future report will describe impacts six years after enrollment. 


The HPOG 1.0 Impact Study is one part of a multipronged research and evaluation strategy 
ACF’s Office of Planning, Research, and Evaluation (OPRE) uses to assess the success of the 
HPOG Program in achieving its goals. The “ACF’s HPOG Research and Evaluation Portfolio” 
textbox on page 8 summarizes this study, its reporting, and related research activities. See 
Appendix A for a description of OPRE’s full portfolio of career pathways evaluations, including 
other research on the HPOG Program. 


This introductory chapter provides the context for this Three-Year Impacts Report. Section 1.1 
describes the Funding Opportunity Announcement (FOA) to which HPOG grantees responded, 
and the textbox that follows describes the program itself in brief. Section 1.2 describes the 
HPOG 1.0 Impact Study, including the history and design of the evaluation, data sources, and 
how HPOG differed from what was otherwise available in the community. Section 1.3 describes 
the characteristics of the study sample. Finally, Section 1.4 closes the chapter with a roadmap 
to the remainder of the report. 


1.1 Career Pathways Framework 


The HPOG Program uses the career pathways framework of postsecondary occupational 
training and education to address the challenge of preparing nontraditional student populations 
who vary in their readiness for training or employment. Programs that operate within a career 


4 Authority for these demonstrations is included in the Affordable Care Act (ACA), Public Law 111-148, 124 Stat. 119, 
March 23, 2010, sect. 5507(a), “Demonstration Projects to Provide Low-Income Individuals with Opportunities for 
Education, Training, and Career Advancement to Address Health Professions Workforce Needs,” adding sect. 
2008(a) to the Social Security Act, 42 U.S.C. 1397g(a). 


> A separate study describes implementation and outcomes of the five Tribal HPOG grantee programs (Meit et al. 
2016). 
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pathways framework generally offer a series of education and training steps within in-demand 
sectors and industries, combined with support services that aim to lead to better job prospects 
and labor market outcomes for participants. 


As outlined in the HPOG FOA (HHS/ACF/OFA 2010), the HPOG model aims to: 


e Target skills and competencies demanded by the healthcare industry. 


e Support career pathways (i.e., offer clearly defined routes that allow participants to build a 
career by advancing through successively higher levels of education and training, exiting 
into employment at multiple possible points). 


e Result in employer- or industry-recognized, portable education credentials (e.g., certificates 
or degrees) and professional certifications and licenses (e.g., a credential awarded by a 
Registered Apprenticeship program). 


e Combine support services with education and occupational training to help participants 
overcome barriers to succeeding in training and finding and keeping a job. 


e Provide training at times and locations that are easily accessible to targeted populations. 


Using discretion within the parameters of the FOA, HPOG grantees designed programs that 
were diverse in context, design, and implementation. The 42 programs included in the HPROG 
1.0 Impact Study were operated by 23 grantees located in 19 states, with varying labor market 
conditions, healthcare labor market structures, and existing community resources. Grantees 
were of various institutional types, including workforce development agencies, postsecondary 
educational institutions, and nonprofit agencies. Each of the grantees made choices about 
eligibility requirements, pre-training activities, healthcare occupational training courses, and 
support services. All of these factors led to potentially different HPOG local program participant 
characteristics, experiences, outcomes, and impacts. 


Local HPOG Programs in Brief 


Local HPOG programs varied in the extent to which they incorporated features of the career pathways framework. The 
National Implementation Evaluation (NIE) and the HPOG 1.0 Short-Term Impacts reports provide substantial descriptions of 
the national HPOG Program itself, including details on variation across local HPOG programs. Sub-sections in this chapter 
provide a broad description of what the HPOG model offered and how that differed from what was otherwise available in the 
community. 


Local HPOG programs enrolled TANF recipients and other low-income individuals with a range of academic and social skills 
and work experience. For those who did not meet the skill requirements for coursework in healthcare, or who needed to 
improve the social skills expected for employment in healthcare, programs provided training in basic academic and “soft” 
skills. For those participants, as well as for others already prepared for occupational training in healthcare, local HPOG 
programs offered training for a variety of healthcare occupations. Some courses were relatively short, usually leading to 
entry-level jobs in the health professions, such as Nurses’ Aide or Phlebotomist. Other courses, such as those for Licensed 
Practical or Registered Nurse, were longer-term, intending to lead to higher-wage jobs. 


A distinctive feature of local HPOG programs was the relatively wide range of available support services intended to help 
participants deal with potential financial and personal challenges to program enrollment and completion. The major types of 
supports included case management; academic supports, such as tutoring; personal and family supports, such as 
assistance with transportation or childcare costs; financial assistance for training and equipment; and employment 
assistance. 
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1.2 HPOG 1.0 Impact Study 


Research on HPOG informs a field of practitioners and policymakers eager for information on 
what is most effective from sector-based, career pathways interventions. 


As shown in Exhibit 1.1, HPOG funding began in September 2010. All of the HPOG 1.0 
grantees were expected to be part of some evaluation research, whether through an ACF- 
sponsored university partnership research study, tribal evaluation, or the HPOG Impact Study. 
Those non-tribal grantees collecting individual-level data as part of another evaluation were not 
required to participate in the Impact Study; the remaining 23 grantees were.® Those 23 grantees 
implemented 42 distinct local HPOG programs in 92 locations across 19 states. 


Exhibit 1.1: HPOG Program and Impact Study Timeline 


MARCH 2013 NOVEMBER 2014 MAY 2018 SUMMER 2021 


Impact Study Impact Study Short-Term Six-Year 
secrEMRER 2010 intake/ — intake ended Impact Report Impact 
randomization released Report 
ie <a began JULY 2014-MARCH 2016 FALL 2019 expected 
— Short-Term Three-Year 
Follow-Up Survey fielded Impact Report 
released 


2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 | 2021 


NOVEMBER 2013-JANUARY 2014 


Staff/Management SEFTEMBEN 2025 


For grantees without 


FEBRUARY 2018-SEPTEMBER 2020 
Six-Year Follow-Up Survey fielded 


Survey fielded 


NOVEMBER 2013-MARCH 2014 
Grantee Survey fielded 


extensions, HPOG 1.0 
funding ended 


FEBRUARY 2015-MAY 2018 
Three-Year Follow-Up Survey fielded 


Notes: Of the 23 HPOG 1.0 grantees in the HPOG 1.0 Impact Study, 18 received a no-cost extension; the timing of extensions varied by 
grantee, but all HPOG 1.0 funding ended by May 31, 2016. 


1.2.1 Evaluation Design 


The HPOG Impact Study uses an experimental evaluation design to assess impacts of 
HPOG 1.0. As shown in Exhibit 1.1, study intake began about 2% years after grants were 
awarded. Beginning in March 2013, local programs randomly assigned eligible applicants to a 
“treatment” group that could access HPOG or to a “control” group that could not. In total, the 
study randomized 13,802 participants.’ This large sample was possible because the FOA 
required all HPOG 1.0 grantees to participate in one or more components of ACF’s career 
pathways research portfolio. 


5 All of the non-tribal HPOG grantees are also part of the National Implementation Evaluation (NIE) of HPOG, and 
they were required to participate in the Impact Study, with a small number of exceptions. In three cases, incumbent 
worker programs were exempted from the Impact Study because of challenges with treatment-control group spillover 
and contamination potential. In addition, one program was not included in the Impact Study because it did not enroll 
any new participants after random assignment began. 


’ Eighty-six participants withdrew from the study sample after random assignment (27 treatment and 59 control). As a 


result, the analytic sample size for this report is 13,716. 
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In 20 of the 23 grantees (which we refer to as “HPOG-only”), for each participant randomized 
into the control group, two participants were randomized into the treatment group (a ratio of 
1:2).8 The remaining three grantees also participated in ACF’s Pathways for Advancing Careers 
and Education (PACE) Study (see Appendix A), an evaluation of career pathways programs not 
limited to healthcare.° In those three grantees (which we refer to as “HPOG/PACE”) the 
randomization ratio was 1:1. 


Unlike other recent research that singles out “promising” programs, the HPOG 1.0 Impact Study 
considers an entire funding stream. The resulting “blended” average assesses whether the 
general HPOG model and the government’s overall investment in that model—across its many 
local implementations—is effective in achieving its goals. 


1.2.2 Data Sources 


The HPOG Impact Study relies on data from varied sources, for measuring the study 
participants’ characteristics, their outcomes, and impacts of the HPOG Program itself. Those 
data sources are as follows: 


e Management information system, called the Performance Reporting System (PRS), which 
was used by all local HPOG programs to collect program data; 


e HPOG baseline survey, which collected data from HPOG study participants at intake 
(supplemental to the PRS baseline data), including a household roster; 


e PACE Basic Information Form (BIF), which collected baseline information from study 
participants in the HPOG/PACE programs; 


e Short-term participant follow-up surveys from the HPOG and PACE Studies, initiated about 
15 months after randomization and completed on average 18 months after randomization; 


e Three-year participant follow-up survey from the HPOG and PACE Studies, (hereafter 
referred to as the “three-year survey”), initiated 36 months after randomization and 
completed on average 40 months after randomization;?° and 


e Employment and earnings data from the National Directory of New Hires (NDNH), covering 
study participants from up to eight quarters prior to their randomization through at least 13 
quarters following it. 


The baseline and NDNH administrative data are available for the full sample. The three-year 
survey had a 73 percent response rate, with 75 percent of those in the treatment group and 70 
percent of those in the control group responding. The survey covers 10,027 of the 13,716 


8 This randomization ratio was implemented mainly for program administration purposes, both to reduce the need for 
programs to over-recruit and to support programs’ cooperation with the evaluation. 

9 As part of the PACE study each of these grantees are evaluated individually. An additional program, the Carerras 
en Salud program at Instituto del Progreso Latino, which implemented one of the Workforce Investment Board of Will 
County’s five local HPOG programs, received HPOG funding and is included in this study but is also evaluated 
separately under PACE. Short-term PACE reports are available on program-specific implementation and impact 
findings, and longer-term PACE reports will be available in the coming months. 

10 The HPOG and PACE studies had slightly different follow-up survey instruments in the short-term. The three-year 


survey was the same for HPOG and PACE. 
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members of the study’s analysis sample.** Additional details of the study sample covered in 
various data sources and how the evaluation handled missing data appear in Appendix B. 


1.2.3. The Experimental Contrast 


In a randomized evaluation such as the HPOG Impact Study, sample members possess 
statistically equivalent observed and unobserved baseline characteristics. Therefore, for impacts 
to arise, the program that the HPOG treatment group is offered needs to be meaningfully 
different from the opportunities available to the control group. This difference is referred to as 
the “experimental contrast.” Understanding how and to what degree treatment conditions differ 
from control conditions is critical to the interpretation of estimated impacts. 


As discussed in the Impact Study’s Short-Term Impacts Report (Peck et al. 2018), the major 
difference between opportunities available to the treatment group and those available to the 
control group was HPOG’s richer support services. Most treatment group members had more 
financial assistance and support services available to them than did the control group members. 
The difference in training course offerings was qualitatively more modest. Most control group 
members had courses available to them that were similar in type, amount, and quality to the 
courses available to the treatment group members. In some communities, treatment and control 
group members may have enrolled in the same training courses. 


1.3. Study Sample Characteristics 


All local HPOG programs recruited and served TANF recipients and other low-income 
individuals. States vary in their eligibility requirements for TANF, and income eligibility limits 
were left to each grantee’s discretion, with most set between 150 and 250 percent of the federal 
poverty level. Grantees sought to recruit applicants who could succeed in healthcare training 
and employment, given some support from the program. Entry requirements considered 
educational background and basic skills, and also screened for training suitability. 


The local programs’ eligibility criteria directly influenced the characteristics of the HPOG 1.0 
Impact Study sample (presented in Exhibit 1.2). Most study participants were “nontraditional” 
postsecondary education students, averaging 32 years old. Most were not married (84 percent) 
and female (89 percent), with dependent children (63 percent). About a quarter of participants 
were already in school or training, and 43 percent were employed when they entered the study. 


11 A subset of respondents who did not answer questions about their education and employment history were 
“skipped out” of the three-year survey. As a result, some of the outcomes based on that survey data have a smaller 
sample size. We discuss our approach to addressing this in our discussion of missing data in Appendix B. 
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Exhibit 1.2: Characteristics of the Study Sample at Baseline 


Sex (%) 
89% Female 


Educational Attainment (%) 
12% Less than 12th grade 

34% High school graduate/GED 
34% Some postsecondary 

19% College degree 


Dependent Children (%) 
63% One or more 


Credential/Degree Completion (%) 


Marital Status (%4) 20% Postsecondary degree/certificate 


see MEL 26% Occupational skills license or 
certification 
ol School Enrollment at Baseline (° 
Averys hoe =? chool Enrollment at Baseline (%) 
g 26% Currently in school 
68% 25 or older 
daoels abit ited S Employment 
Borin Cre Dale ee 2.4 Quarters employed in year prior 
to baseline 
Race and Ethnicity (%) 43% Currently working 


24% Hispanic/Latino of any race 

34% Black/African American, non-Hispanic 
36% White/Caucasian, non-Hispanic 

7% Other, non-Hispanic 


Earnings in Past Year ($) 


$2,317 Average quarterly earnings 
in year prior to baseline 


Expect to Participate in HPOG (%) 
74% Full-time 
26% Part-time 


Barriers to School/Work 
(fairly often or very often) (%) 


17% Childcare arrangements 
18% Transportation 


Public Assistance Use 12% Participant's illness or health 
56% Receiving SNAP/WIC condition 


12% Receiving TANF 1% Alcohol or drug use 


OO SQ006°0 
8 O000 O 


Notes: See Appendix B for a detailed description of data sources and rates of missing data. 
Sources: HPOG Performance Reporting System (PRS) and PACE Basic Information Form (BIF) 


Given the program’s eligibility requirements for personal and work-related skills and motivation, 
the control group seemed likely to persevere in its desire to complete occupational education 
notwithstanding their exclusion from HPOG. These traits flag that the sample was relatively 
motivated in terms of school, training, and work. Because these traits describe the entire study 
sample—that is, both treatment and control group—they set a high bar for the HROG Program 
to improve the treatment group’s educational progress and earnings over the control group. 
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1.4 


1. INTRODUCTION 


Organization of the Report 


The remainder of the report proceeds as follows: 


Chapter 2: The HPOG Program Logic Model and Early Impacts describes the HPOG 
career pathways logic model, and how it shapes both the evaluation design and its results. 
We summarize the study’s early impacts and state the three-year impact analysis’s research 
questions. 


Chapter 3: Impacts on Educational Progress reports the overall impact of HROG on the 
confirmatory outcome of training completion and on other educational outcomes. 


Chapter 4: Impacts on Labor Market Outcomes reports the overall impact of HROG on 
the second confirmatory outcome of earnings (average in the 12th and 13th quarters after 
random assignment) and on other labor market outcomes. 


Chapter 5: Impacts on General Well-Being reports impacts on income, public assistance 
receipt, and economic and psychological well-being outcomes. 


Chapter 6: Impacts on Subgroups reports impacts on confirmatory and secondary 
outcomes for demographic, policy-relevant, and public assistance-defined subgroups. 


Chapter 7: Impacts on Child Development & Well-Being reports impacts on confirmatory, 
secondary, and selected exploratory outcomes for the subsample of study participants with 
dependent children. 


Chapter 8: Patterns of Training & Employment in the HPOG Treatment Group 
describes the education and employment experiences of HPOG treatment group members 
during the three years after enrollment in the program. 


Chapter 9: Discussion & Conclusion summarizes the three-year findings and their 
implications for policy and practice. 


Appendix material includes results of alternative specifications or additional details not 
included in the main body of the report. The following are provided in a separate volume: 


- Appendix A: HPOG as Part of OPRE’s Career Pathways Research Portfolio 
- Appendix B: Sample Sizes and Missing Data 

- Appendix C: Supplemental Materials for Chapter 3 

- Appendix D: Supplemental Materials for Chapter 4 

- Appendix E: Supplemental Materials for Chapter 5 

- Appendix F: Supplemental Materials for Chapter 6 

- Appendix G: Supplemental Materials for Chapter 7 


- Appendix H: Supplemental Materials for Chapter 8 
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This chapter presents the HPOG Program logic model, summarizes the study’s short-term 
impacts (as reported in the study’s Short-Term Impacts Report), and then suggests implications 
for the three-year follow-up. 


2.1 Logic Model 


The study’s streamlined logic model (Exhibit 2.1) shows how the program inputs, activities, and 
outputs can lead to expected outcomes.” In brief, the model suggests how program context, 
administration, and content associate with anticipated education, training, and other program 
activities. These are expected to lead to basic program outputs, which are intended to influence 
various outcomes. 


Exhibit 2.1: HPOG Career Pathways Framework Logic Model (Streamlined) 
OES ACTIVITIES OUTPUTS OUTCOMES 


* Program funding + Intake, enrollment * Basic educational upgrades + Educational progress 
* Program planning, design * Assessments * Training course(s) completion (initial and continued) 
+ Administration * Career pathways-informed * Occupational skills * Credentials earned 
+ Implementation strategies education, training + Work-readiness skills + Employment (overall, in 
(contextualized, accelerated, * Career plan healthcare) 
and flexible delivery) * Earnings 


* Support services + Job quality 
+ Employer connections * Public assistance 
* Family and child well-being 


CONTEXT & POPULATION 


The logic model hypothesizes a path from the HPOG Program through its outputs to short- and 
long-term outcomes. Specifically, program activities such as support services increase the 
likelihood that participants will complete the program. Program completion and associated 
credential attainment are expected to lead to labor market impacts. Favorable labor market 
impacts are expected to result in improved family and child well-being. 


The logic model also recognizes that certain factors—including local context and population 
characteristics—are beyond the control of program designers and operators. For instance, local 
labor market conditions likely contribute to employment outcomes, and population 
characteristics likely influence the training programs and services provided. However, these 
factors are the same for both treatment and control group and so do not affect impact estimates. 


2.2 Short-Term Impacts 


This section summarizes the key findings from the study’s Short-Term Impacts Report, a 
snapshot taken at approximately 15-18 months after study enrollment. 


12 A more fully elaborated logic model appears in other project documents including the Short-Term Impacts Report 
(Peck et al. 2018) the, Impact Study Design Report (Peck et al. 2014), and elsewhere in implementation reports (e.g., 
Werner et al. 2018). 
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Access to HPOG increased study participants’ enrollment in training: 71 percent of the 
treatment group and 62 percent of the control group enrolled in training as of the short-term 
follow-up, a difference that represents about a 15 percent improvement (Exhibit 2.2). This 
difference stemmed from greater enrollment in occupational classes (as opposed to credit 
classes, ESL classes, or basic skills classes; not shown). 


Exhibit 2.2: Summary of HPOG Impacts on Participation in Training and Services at 
Short-Term Follow-Up 


+9.3° 
+92” 
13.1" +12.4- 
71.4 39.3 


Percentage of Study Participants (%) 


Enrolled in Received any Received any Received any 
Training or Related Career Support Academic Support Other Support 
Activities Services Services Services 


Control Group isa Treatment Group Yi Relative Impact 


Notes: Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
The more asterisks, the more likely the finding is not due to chance. 

Sample Sizes and Sources: 

Enrolled in Training or Related Activities, and Received Any Career Support Services: Treatment: 6,801. Control: 3,649. HPOG and PACE 
short-term follow-up surveys. 

Received Any Academic Support Services, and Received Any Other Support Services: Treatment: 5,566. Control: 2,525. HPOG short-term 
follow-up survey. 


The treatment group reported substantially greater levels of receipt of support services than the 
control group: about 9 percentage points more reported receiving academic support services, 
about 13 percentage points more reported receiving career support services, and about 12 
percentage points more reported receiving other support services such as help arranging 
supports, counseling services, non-cash incentives, or emergency assistance services. These 
differences represent between about a 20 percent and 50 percent increase of services received 
by the treatment group relative to the control group. 


As shown in Exhibit 2.3, HPOG participants made statistically significant educational progress, 
the Impact Study’s short-term confirmatory outcome.** Educational progress is defined as 
Ongoing enrollment in training or completion of training as of the short-term follow-up survey. 
About 60 percent of the control group versus 68 percent of the treatment group had made 


13 Definitions of confirmatory and secondary outcomes appear in the textbox “Impact Analysis Data and Analytic 
Methods” on page 23. 
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educational progress. That 7 percentage point impact reflects a 12 percent improvement of the 
treatment group over the control group. Based on this finding, the Short-Term Impacts Report 
concluded that HPOG had achieved its short-term goal of increasing educational progress. 


HPOG also had favorable impacts on secondary outcomes in the short-term, such as healthcare 
sector employment, job quality (defined as access to employer-sponsored health insurance), 
and earnings, as shown in Exhibit 2.3. Although there was no impact on employment in general, 
employment was channeled into healthcare: the treatment group’s healthcare sector 
employment was 11 percentage points greater than that of the control group, reflecting a 
relatively large 27 percent increase. This increase in healthcare sector employment was an 
explicit goal of HPOG. Further, the treatment group earned $137 more than the control group in 
the fifth follow-up quarter, a relatively small 4 percent increase. 


Exhibit 2.3: Summary of HPOG Impacts on Confirmatory and Secondary Outcomes at 
Short-Term Follow-Up 


SJ 

wo 

e 

s 

S 
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© 69.7 ui S a 

2 EG so 
Educational Employment Employed Job Offers Public Earnings 

Progress in Q5 in Healthcare Health Assistance in Q5 


Insurance (TANF) Receipt 


Control Group [ Treatment Group i Relative Impact 


Notes: Statistical significance levels for one-sided tests are indicated with hashtags, as follows: ### = 1 percent; ## = 5 percent; # = 10 percent. 
The more hashtags, the more likely the finding is not due to chance. 

Sample Sizes and Sources: 

Educational Progress, and Employed in Healthcare: Treatment: 6,801. Control: 3,649. HPOG and PACE short-term follow-up surveys. 
Employment in Q5, and Earnings in Q5: Treatment: 8,673. Control: 5,044. National Directory of New Hires. 

Job Offers Health Insurance, and Public Assistance (TANF) Receipt: Treatment: 5,566. Control: 2,525. HPOG short-term follow-up survey. 


In addition to reporting on these overall impacts of HPOG, the Short-Term Impacts Report 
examines a wide variety of disaggregated impacts, including impacts defined by various 
subgroups of individuals and type of grantee, as well as incremental impacts that derive from 
selected program designs and implementation choices. The report concludes that HPOG was 
generally effective for people regardless of their demographic characteristics. Those people who 


14 The Short-Term Impacts Report as originally released in May 2018 contained an error in the subgroup impact 
estimates. A corrected report is available as of November 2019 at https://www.acf.hhs.gov/opre/resource/health- 
profession-opportunity-grants-hpog-10-impact-study-interim-report-implementation-short-term-impacts. The text here 
reflects the corrected results, which modified the conclusions of the subgroup analysis from those originally reported. 
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were receiving TANF, SNAP, or WIC at baseline had larger favorable impacts on educational 
progress than those who were receiving no public assistance at baseline. Those who came to 
HPOG not yet enrolled in education or training also saw greater gains in healthcare sector 
employment and job quality as well as a reduction in TANF receipt, relative to their control group 
counterparts. Those who were not employed at baseline or who entered with some college or a 
degree also saw greater gains in job quality than those who were employed or entered with a 
high school degree or less, respectively. 


Although there were not differential subgroup impacts on educational progress based on 
presence of barriers to participation or work, those without any barriers at baseline showed 
larger impacts on earnings, relative to their control group counterparts. 


The HPOG logic model implies that the combined evidence of increased enrollment in training 
and receipt of support services should affect a variety of outcomes, including those related to 
training, the labor market, and general well-being. As a result, the Short-Term Impacts Report 
concluded that HPOG has the potential to generate longer-term effects on education- and 
employment-related outcomes. 


2.3. Confirmatory Outcomes and Research Questions for the Three-Year 
Analysis 


For this Three-Year Impacts Report, we pre-specified two confirmatory outcomes in our Three- 
Year Analysis Plan (Litwok et al. 2018). In line with the ACF Evaluation Policy’s principle of 
transparency, the Analysis Plan was published and registered prior to conducting any 
analyses.*°:76 


The confirmatory measure in the Short-Term Impacts Report was educational progress, which 
was defined as having completed training or still being enrolled in training. We defined 
educational progress in this way because we expected many participants to remain enrolled in 
training as of the short-term follow-up. As of three years after random assignment, we expected 
that most study participants—including those who were in short- and longer-term educational 
programs or trainings—would have completed at least one education or training course. As a 
result, we committed to training completion as the appropriate measure of educational progress 
at the three-year follow-up. 


The shift from educational progress to training completion also signals that study participants 
were expected to have transitioned to the labor market, making their earnings an especially 
important indicator of HPOG’s success at this time. As a result, we also committed to earnings 
as a second confirmatory outcome. As a measure, earnings captures changes to study 
participants’ employment, wages, and hours worked. As such, earnings is a comprehensive 
measure of labor market success. 


15 Registration of the three-year analysis is available here: https://osf.io/z986c/. 
16 ACF’s Evaluation Policy is available here: https://www.acf.hhs.gov/opre/resource/acf-evaluation-policy. 
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We pre-specified both of these confirmatory outcomes to limit the number of statistical tests 
from which we would draw conclusions.*’ As in the Short-Term Impacts Report, we refer to 
confirmatory outcomes by either their variable description or their domain. Exhibit 2.4 
summarizes the confirmatory outcomes in each domain and their definitions, both in this report 
and the Short-Term Impacts Report. 


Exhibit 2.4: Confirmatory Outcomes 
Domain Measure Description Data Source(s) 
Short-Term Impacts Report 


Educational progress Training completion or ongoing enrollment in training HPOG and PACE short-term 


follow-up surveys 


Three-Year Impacts Report 
Educational progress Training completion Three-year survey 


Earnings Average quarterly earnings during the 12th and 13th quarters © NDNH 
after random assignment 


Notes: The HPOG and PACE studies had slightly different follow-up survey instruments in the short-term. The three-year survey was the same 
for HPOG and PACE. 


2.3.1 Research Questions for Three-Year Follow-Up 


With HPOG’s logic model and the conclusions of the Short-Term Impacts Report in mind, the 
analysis in this latest report approximately three years after participants entered the study 
examines the following research questions: 


e What impacts do the locally implemented HPOG programs as a group have on the 
outcomes of participants and their families? 


To what extent do those impacts vary across selected subpopulations? 


To what extent do the education and employment experiences of HPOG participants over 
time suggest that they are following a career pathway? 


Chapters 3, 4, 5, and 7 address the first (overall impacts) question; Chapters 6 and 7 address 
the second (subgroup impacts) question; and Chapter 8 addresses the third (descriptive) 
question. 


1” See Section 2.2.1 of the Three-Year Analysis Plan (Litwok et al. 2018) for more detailed discussion on this issue. 
Although we have two confirmatory outcomes, because they are in different domains, we make no statistical 
adjustments, per What Work Clearinghouse standards (ED/IES 2017). 
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Impact Analysis Data and Analytic Methods 


The chapters that follow report impacts of the average effect of being offered access to the HPOG Program. This textbox 
summarizes outcome classifications, data sources, and analytic methods in brief. 


Outcome Classifications 

The impact analysis distinguishes among three types of outcomes: confirmatory, secondary, and exploratory. The 
confirmatory outcomes are the main indicators of the extent to which the program is making progress toward its goals. 
Secondary outcomes are additional important outcomes identified in the HPOG logic model. Exploratory outcomes are 
of two types: (1) outcomes of interest that may be affected by the program but are not identified in the logic model; and (2) 
alternative specifications of confirmatory and secondary outcomes. 


The logic model hypothesizes that HPOG will have a favorable impact on each of the confirmatory and secondary outcomes 
by three years after random assignment. Because these hypotheses are directional, we conduct one-sided hypothesis tests 
for confirmatory and secondary impacts. For one-sided tests, failure to reject the null hypothesis indicates that there is no 
evidence that HPOG had a favorable impact for that outcome. 


We do not necessarily have directional hypotheses for impacts on exploratory outcomes, so we conduct two-sided 
hypothesis tests. We also estimate the impact for selected outcomes by subgroup and test for differences in impacts 
between subgroups. The subgroup analyses are exploratory, so all of the subgroup tests—both the impacts for each 
subgroup and differences between subgroups—are conducted using two-sided hypothesis tests. 


Data Sources 

We began administering a follow-up survey of study participants approximately three years after their random assignment. 
The three-year survey captures data to measure impacts on key intermediate-term program outcomes. The survey 
captures information on participants’ education and employment history; credentials earned; wages, hours worked, fringe 
benefits, and other indicators of job quality; school loans and other educational assistance; 21st century skills; and overall 
indicators of financial well-being. In addition, for participants with children, the survey captures information about parenting 
practices and the educational experiences of children. 


We used data from the National Directory of New Hires (NDNH) to construct outcomes of earnings and employment at 
about three years after randomization, as well as baseline measures of the same constructs. The NDNH collects individual 
quarterly earnings from state Unemployment Insurance (UI) records, augmented with data from some employers not 
included in the UI program (e.g., the federal government). Other earnings and employment types, in particular self- 
employment income, are not reported to the NDNH. Generally, participants missing earnings in NDNH in a particular quarter 
were not employed in covered jobs in that quarter. However, a small proportion may have been employed in covered jobs 
but the administrative records did not match. 


Impact Analysis Methods 

Because the treatment and control group members were assigned to their groups randomly, the only systematic difference 
between the groups is that the treatment group members were offered the opportunity to participate in HPOG. As is 
standard practice, the study confirmed analytically that the two groups are otherwise similar (see Harvill, Moulton, and Peck 
2015). 


The difference in mean outcomes between the treatment group members and the control group is an estimate of the HPOG 
Program’s “impact.” With random assignment, we can infer that any difference between those mean outcomes that is 
Statistically significantly different from zero was caused by the offer of HPOG services. Because we randomized the offer of 
access to HPOG, this impact analysis provides an estimate of the impact of that offer. It ignores whether participants 
actually took up that offer (although most, 96 percent, did). 


To estimate the impact on our designated outcomes, we use a multi-level regression model that ensures that standard 
errors are calculated accurately for individuals within programs. For additional information related to the computation and 
presentation of the report's outcome measurement, data manipulation, and impact estimation procedures, see the Three- 
Year Analysis Plan (Litwok et al. 2018). 
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How to Read the Impact Exhibits in This Report 


The exhibits in the chapters that follow report on the impact of HPOG using a consistent format. Here we describe the 
appropriate interpretation of information in these tables. The sample table below presents the impact of HPOG on three 
measures of educational progress from Exhibit 3.1. The table reports the mean level of each outcome for both the treatment 
and control groups. The “Level of Evidence” column reports whether outcomes are confirmatory, secondary, or exploratory, 
as defined in the preceding textbox. 


For each outcome the difference between the two means is the impact of being offered HPOG services, estimated using 
multiple regression. The “Treatment Group Mean” and “Control Group Mean” columns report regression-adjusted means. 
The table’s “Impact” column shows that the treatment group was about 13 percentage points more likely than the control 
group to have completed training. We also report relative impact in the right-most column, which is the impact divided by the 
control group mean. The relative impact offers a sense of how “big” or “small” the impact is, at least relative to the control 
group’s level. 


Impacts marked with one or more symbols are statistically significant, indicating that it is unlikely that these impacts are due 
to chance. Confirmatory and secondary tests are one-sided (with statistical significance indicated by hashtags), tests we 
use because we have a directional hypothesis for these impacts. In comparison, exploratory tests use a two-sided test (with 
Statistical significance indicated by asterisks), a test we use because we do not have a directional hypothesis. Three, two, 
or one symbol, respectively, corresponds to whether the impact is statistically significant at the 1 percent, 5 percent, or 10 
percent level. The more symbols, the less likely the finding is due to chance. 


Sample Table. Overall Impacts on Educational Progress Outcomes 


Level of Treatment Control Relative 
Outcome Evidence Group Mean Group Mean Impact Impact 
Training completion (%)@ Confirmatory 75.2 62.7 12, 5### 19.9 
Obtained certificate, license, or Exploratory 58.1 45.4 d2 28.0 
credential (%)4 
Confidence in career knowledge Secondary 3.40 3.37 0.03## 0.8 
(range is 1 to 4)» 


Notes: Confirmatory and secondary findings use a one-sided hypothesis test, and exploratory findings use a two-sided hypothesis test. 
Statistical significance levels for one-sided tests are indicated with hashtags, as follows: ### = 1 percent; ## =5 percent; # = 10 percent. 
Statistical significance levels for two-sided tests are indicated with asterisks, as follows: ** = 1 percent; ** = 5 percent; * = 10 percent. 
Sample Sizes and Sources: 

@ Treatment: 6,027. Control: 3,305. Three-year survey. 

» Treatment: 6,476. Control: 3,551. Three-year survey. 


Appendix B provides additional detail on the construction of measures, including outcomes, covariates, and subgroup 
identifiers. Appendices C through G contain analytic results that include standard errors, confidence intervals, and post-hoc 
minimum detectable effect calculations, instructions on how to interpret these additional numbers, and impact estimates for 
additional exploratory outcomes. 
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Summary of Key Findings: Educational Progress 


e HPOG increased educational progress, defined as completing training. As of the three-year survey 75 
percent of the treatment group had made educational progress, compared to 63 percent of the control 
group. The impact of about 13 percentage points represents a 20 percent relative increase. 


e HPOG increased average training duration, but it did not increase receipt of longer-term college 
certificates or degrees. The treatment group was more likely to obtain a certificate, license, or credential; 
complete a diploma or certificate for regular college classes; or receive a vocational training certificate. 


e HPOG increased self-reported confidence in career knowledge. The increase of 0.03 points on a four- 
point scale is small relative to the control group. HPOG also increased self-reported progress toward long- 
term educational goals by 4 percentage points, a relative increase of 6 percent. 


This chapter reports HPOG’s impact on educational progress. The analysis assesses the extent 
to which HPOG affects enrollment in and completion of training, confidence in career 
knowledge, and perception of progress toward long-term educational goals. These outcomes 
reflect key features of HPOG’s logic model, which predicts that impacts within this domain will 
lead to impacts on labor market outcomes such as employment and earnings. 


3.1 Overall Impacts on Educational Progress Outcomes 
e HPOG increased educational progress, defined as having completed training. 


As of the three-year survey, 75 percent of the treatment group had completed training, 
compared to 63 percent of the control group. The difference of 13 percentage points represents 
a nearly 20 percent relative improvement over the control group. Exhibit 3.1 reports the impact 
results for training completion and other measures of educational progress defined in the 
textbox on the next page. 


HPOG increased three of the four measures that comprise the study’s primary measure of 
educational progress. HPOG led to a 13 percentage point increase in obtaining a certificate, 
license, or credential (58 percent in the treatment group versus 45 percent in the control group). 
The treatment group also experienced an increase of 6 percentage points on completion of a 
diploma or certificate for completing regular college classes (18 percent in the treatment group 
versus 11 percent in the control group). Finally, there was an 8 percentage point impact on 
receipt of a vocational training certificate (22 percent in the treatment group versus 14 percent in 
the control group). Each of these increases was sizeable relative to the control group. 
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Overview of Outcome Measures: Educational Progress 


The outcomes in this chapter are based on participant responses to the three-year survey. The confirmatory 
outcome in this chapter, which reflects whether the HPOG Program is successful in achieving its goals, is training 
completion. Our set of secondary and exploratory outcomes provide additional evidence regarding HPOG’s 
impact on educational progress. We define confidence in career knowledge as a key secondary measure, which 
captures the added psycho-social benefit that is predicted to materialize in the longer-term as participants make 
educational progress and embark on a career in their chosen field. We also define a series of exploratory 
outcomes, which measure different features of education and training. 


Outcome Outcome Type Variable Description 
Training completion Confirmatory | Measures whether respondent achieved any of the following since 
randomization: 
e Obtained a professional, state, or industry certificate, license, or 
credential 
e Completed a diploma or certificate for completing regular college 
classes 


e Received a diploma or certificate from a school for completing 
vocational training 

e Completed a degree (associate's or bachelor’s) 

The next four variables each represent one of these elements. 


Obtained certificate, license, Exploratory Obtained a professional certificate, license, or credential since 


or credential randomization 

Completed diploma or Exploratory Completed a diploma or certificate for completing regular college classes 
certificate for regular college (excluding associate's or bachelor’s degrees) since randomization 
classes 


Received vocational training Exploratory Received a vocational training diploma or certificate since randomization 
certificate 


Completed college degree Exploratory Completed college degree, either associate's or bachelor’s, since 
randomization 


Months of full-time equivalent Exploratory Total months of full-time equivalent (FTE) enrollment in education or training 
enrollment since randomization. One month of full-time enrollment is equal to 1 FTE 
month; one month of part-time enrollment is equal to 0.5 FTE months. 


Earned 2 1 year college Exploratory Earned a degree or certificate for completing regular college classes 
certificate or degree requiring at least a full year of credit since randomization 
Currently enrolled in training Exploratory Enrolled in training at the time of the three-year follow-up 
Confidence in career Secondary Scale measuring confidence in career knowledge, scale ranges from 1 to 4: 
knowledge 1. I'm not sure how to accurately assess my abilities and challenges; 
2. | know how to make a plan that will help me achieve my goals for the next 
five years; 


3. | know how to get help from staff and teachers with an issues that might 
arise when | am at school; 

4. I'm not sure what type of job is best for me; 

5. | know the type of employer | want to work for; 

6. | know the occupation | want to be in; 


7. I'm not sure what kind of education and training program is best for me. 
Perception of progress Exploratory Strongly or somewhat agrees with the statement “I am making progress 
toward long-range towards my long-range educational goals” 
educational goals 
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Exhibit 3.1: Overall Impacts on Educational Progress 


Treatment Control Relative 
Outcome Level of Evidence GroupMean Group Mean Impact Impact 
Training completion (%)@ Confirmatory 75.2 62.7 12, 5### 19.9 
Obtained certificate, license, or Exploratory 58.1 45.4 12.0 28.0 
credential (%)a 
Completed diploma or certificate for Exploratory 17.7 11.2 6.4** 57.4 
regular college classes (%)@ 
Received vocational training Exploratory 21.7 14.0 Ll 55.4 
certificate (%)a 
Completed college degree (%)@ Exploratory 23.0 23.0 -0.0 -0.1 
Months of full-time equivalent Exploratory 7.6 6.8 0.8" 11.1 
enrollmenta 
Earned 2 1 year college certificate or Exploratory 27.0 25.5 15 5.8 
degree@ 
Currently enrolled in training (%)@ Exploratory 13.0 13.3 -0.2 -1.6 
Confidence in career knowledge Secondary 3.40 3.37 0.03## 0.8 
(range is 1 to 4) 
Perception of progress toward long-term — Exploratory 75.2 71.1 4,2" 5.9 


educational goals (%)» 


Notes: Confirmatory and secondary findings use a one-sided hypothesis test, and exploratory findings use a two-sided hypothesis test. 
Statistical significance levels for one-sided tests are indicated with hashtags, as follows: *** = 1 percent; ##* = 5 percent; * = 10 percent. 
Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Sample Sizes and Sources: 

a Treatment: 6,027. Control: 3,305. Three-year survey. 

> Treatment: 6,476. Control: 3,551. Three-year survey. 


e HPOG increased average training duration, but did not increase receipt of longer-term 
college certificates or degrees. 


In addition to increasing completion of training, HPOG also led to more months of training. On 
average, the treatment group completed 7.6 full-time equivalent months of training as of the 
three-year survey, an increase of 0.8 months (about 24 days) over the control group. 


Despite this increase in training duration, HPOG did not lead to an increase in receipt of longer- 
term credentials, as of the three-year follow-up. For example, there was no detectable impact on 
completion of a college degree (just under a quarter of both the treatment and control groups), 
which includes associate’s degrees or higher. Similarly, there was no detectable impact ona 
broader outcome that includes degrees or certificates that require a year or more of college 
credit (just over a quarter of both groups). 


These results indicate that HPOG was successful in increasing both the duration of training and 
receipt of professional and vocational certificates. However, HPOG training did not lead to an 
increase in longer-term degrees for the sample as a whole. It is possible that some study 
participants may have been enrolled in longer-term programs that extend beyond the three 
years examined in this report. However, only about 13 percent of the sample was still enrolled in 
education or training at the three-year follow-up (with no statistically significant difference 
between the treatment and control group). As a result, we would not expect to a see substantial 
change in the impact on degree receipt from those still enrolled. 


e HPOG increased the treatment group’s self-assessed confidence in career knowledge 
and perception of progress toward long-term educational goals. 
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In addition to higher rates of educational progress, the treatment group reported higher levels of 
self-assessed career knowledge and progress toward long-term educational goals. On a scale 
from 1 to 4, the treatment group reported a score of 3.40 in confidence in career knowledge, 
versus a score of 3.37 for the control group. These scores imply that, on average, both groups 
answer between “somewhat agree” and “strongly agree” to statements about their ability to 
assess their own abilities, make a plan to achieve their goals, get help from staff and teachers, 
find an occupation and employer, and determine the best education or training program for 
themselves. This impact of 0.03 points is substantively small.*® 


In addition, 75 percent of the treatment group reported progress toward their long-term 
educational goals, compared to 71 percent of the control group. This 4 percentage point 
difference represents a 6 percent relative increase. 


3.2 Summary 


As described in the logic model, we expect that HPOG would have led to an increase in 
educational progress in the three years after random assignment. Indeed, we found that HPOG 
increased training completion, which is one of the two confirmatory outcomes examined in this 
report. These results are broadly consistent with the findings in the Short-Term Impacts Report, 
which also found that HPOG increased educational progress, defined then as enrollment in or 
completion of training as of about 15-18 months after random assignment. 


A sizeable share of the control group still accessed training despite not having access to HPOG. 
Nearly two thirds of the control group completed training, suggesting that local HPOG programs 
operated in training-rich environments and targeted very motivated participants. Control group 
members also reported high scores on self-assessed confidence in career knowledge and 
perception of progress toward their educational goals, consistent with the high level of training 
access. 


18 We do not discuss the magnitude of the relative impact for this measure because it is difficult to interpret the 
relative impact for an ordinal measure. 
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Summary of Key Findings: Labor Market 


e HPOG had no detectable impact on earnings and a small impact on overall employment. Both the 
treatment and control group earned an average of approximately $5,000 over those same quarters. 
Administrative earnings data show 82 percent of the treatment group was employed in the 12th or 13th 
quarter after random assignment, a statistically significant improvement of 1 percentage point over the 
control group. 


e HPOG increased healthcare employment and improved employment conditions. According to the 
three-year survey, 56 percent of the treatment group was employed in healthcare, compared to 44 percent 
of the control group. The treatment group was also more likely to work at a job that offered health insurance 
and report that they worked in a supportive work environment. 


e HPOG improved career progress, defined as both completing training and having earnings growth. 
Forty-three (43) percent of the treatment group made career progress, versus 36 percent of the control 
group. Exploratory analyses suggest that this impact is driven by increases in training completion. 


In this chapter we report HPOG’s impact on labor market outcomes. These outcomes include 
measures of general earnings and employment and other outcomes reflecting the conditions of 
employment. We also report on career progress, or the extent to which individuals experienced 
both educational progress and earnings growth. These outcomes represent the next step in 
HPOG'’s logic model after initial education or training. The favorable impacts on training 
completion in Chapter 3 set the foundation for possible favorable impacts on labor market 
outcomes. 


4.1 Impacts on Earnings and Employment 


e HPOG had no detectable impact on earnings in the 12th and 13th quarters after 
randomization. 


According to administrative earnings data from the NDNH, both the treatment and control 
groups earned about $5,000 per quarter ($5,039 for the treatment group, $4,997 for the control 
group; Exhibit 4.1). These earnings correspond to annual earnings of approximately $20,000.’9 
Our analysis concludes that, if HPOG had any impact on earnings, it was smaller than the 
evaluation could detect (no larger than $170 in the 12th and 13th quarters).”° 


19 In Appendix D, we use survey data as a sensitivity analysis to investigate whether we can detect any changes to 
weekly earnings, hours worked, or both. The analysis reveals a positive impact of $15 on weekly earnings and no 
detectable impact on weekly hours or the hourly wage. These findings are not inconsistent with the NDNH findings: 
an impact of $42 per quarter is equivalent to an impact of about $3 per week, which is within the 90 percent 
confidence interval of survey-reported earnings. We also estimate the non-experimental impact on the hourly wage. 


20 See Appendix D for standard errors and confidence intervals that correspond to the findings in this chapter. 
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Overview of Outcome Measures: Labor Market 


The outcomes in this chapter are based on two sources: participant responses to the three-year survey and 
administrative data on quarterly earnings from the NDNH. We selected earnings in the 12th and 13th quarters 
as the confirmatory outcome for the three-year follow-up as the key measure by which we assess whether HPOG 
improved labor market outcomes. We chose earnings because it captures changes to employment, wages, and 
hours worked. In addition to earnings, we define four secondary outcomes that are also important measures of 
HPOG'’s impact on labor market outcomes: (1) employment in the 12th or 13th quarter measures whether 
participants were employed in either of these quarters; (2) current or most recent job in healthcare is a 
measure of whether HPOG led to increased employment in the healthcare sector; (3) current or most recent job 
offers health insurance is a measure of job quality to assess whether participants are employed in jobs with 
benefits; and (4) training completion and earnings growth is a measure of career progress, which measures the 
extent to which participants both completed training and experienced earnings growth. 


Outcome (Source) Outcome Type Variable Description 

Earnings (NDNH) 

Earnings in 12th and 13th — Confirmatory Average quarterly earnings during the 12th and 13th quarters after random 
quarters assignment 

Earnings in each of Exploratory Earnings in each of the quarters: quarter of randomization through quarter 
quarters 0 through 13 13 


Employment (NDNH) 


Employment in 12th or 13th Secondary Employed in either the 12th or 13th quarter after the quarter of 

quarter randomization 

Employment in each of Exploratory Employment in each of the quarters: quarter of randomization through 
quarters 0 through 13 quarter 13 

Employment Conditions (Survey) 

Current or most recent job Secondary Current or most recent job is in healthcare 

is in healthcare 

Current or most recent job Secondary Current or most recent job offers health insurance 

offers health insurance 

Supportive working Exploratory Currently employed in a job offering a supportive working environment, as 
environment measured by access to health insurance benefits, sick leave, promotion 


potential, supervisor support, and flexible work schedule 
Career Progress (NDNH & Survey) 


Training completion and Secondary Completed training and experienced average earnings growth of at least 
earnings growth $20 per quarter between 5th quarter and 13th quarter after randomization@ 
Notes: 


a We determined the wage growth threshold by using the earnings observed in our sample and typical wage growth observed in the general 
population. For wage growth, we used median one-year wage growth rates from the Current Population Survey between January 2011 and 
July 2018 as reported by the Atlanta Federal Reserve (see https:/Awww. frbatlanta.org/chcs/wage-qrowth-tracker.aspx?panel=1). Overall, the 
average growth rate was 2.71%. The growth rate varied across subgroups and occupations as follows: high school diploma or less: 2.48%; 
associate's degree: 2.6%; low-skill occupations: 1.9%; education and health industries: 2.3%. Overall, this means that a one-year growth 
rate of between 2% and 2.5% is reasonable, which corresponds to a 0.55% growth in quarterly earnings. We then multiply 0.55% by the 
average earnings in the 5th quarter after randomization observed in our sample ($3,671), which equals approximately $20 per quarter. 
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e HPOG slightly increased overall employment in the 12th and 13th quarters after 
randomization. 


Exhibit 4.1 also reports that 82 percent of treatment group members were employed at some 
time during the 12th or 13th quarters after randomization, compared to 81 percent of control 

group members. Though this 1 percentage point impact is small in magnitude, it represents a 
statistically significant improvement over a very high rate of employment in the control group. 


Exhibit 4.1: Impacts on Earnings and Employment 


Level of Treatment Control Relative 
Outcome Evidence Group Mean Group Mean Impact Impact 
Average quarterly earnings during the Confirmatory 5,039 4,997 42 0.8 
12th and 13th quarters ($) 
Employment in 12th or 13th quarter Secondary 82.0 81.0 1.0# 1.2 
(%) 


Notes: Confirmatory and secondary findings use a one-sided hypothesis test, and exploratory findings use a two-sided hypothesis test. 
Statistical significance levels for one-sided tests are indicated with hashtags, as follows: *** = 1 percent; ## = 5 percent; * = 10 percent. 
Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Sample Sizes and Sources: 

Treatment: 8,672. Control: 5,044. National Directory of New Hires. 


Exhibits 4.2 and 4.3 show how both earnings and employment, respectively, increased 
substantially over time for both the treatment and control groups. As is common in job training 
programs, just after randomization the treatment group had lower levels of earnings and 
employment than the control group, likely due to the treatment group’s greater participation in 
training. 


The early negative earnings impacts are not statistically significant after the third quarter. There 
were generally no detectable earnings impacts—with the exception of in the seventh follow-up 
quarter, where the impact of $140 is statistically significant.** Similarly, after the third quarter, 
the impacts on employment are not statistically significant; toward the end of the follow-up 
period (Q11), the impacts become positive, although small. 


In addition to analyzing quarterly impacts, we examined whether the trend for earnings or 
employment differed between the treatment and control groups. That analysis found no 
statistically significant differences in the slopes of the trend lines (see Appendix D). 


21 The Short-Term Impacts Report found a statistically significant impact on quarterly earnings in Q5 of $137, with the 
treatment group earning $3,482 and the control group earning $3,345. Two things have changed from that report 
First, NDNH data were updated, which increased average earnings levels and slightly changed the Q5-reported 
impact (to $128). Second, in the Short-Term Impacts Report, Q5 earnings was a secondary outcome, for which 
statistical significance was determined using a one-tailed test. Applying that one-tailed test, both impact estimates 
would be statistically significant (.05 < p < .10). In this report, Q5 earnings are an exploratory outcome, for which 
statistical significance is determined using a two-tailed test. If we would apply the two-tailed test used in this Three- 
Year Impacts Report, both the earlier reported estimated impact of $137 and the currently reported impact of $128 
would not be statistically significant (o > .10). 
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Exhibit 4.2: Impacts on Quarterly Earnings Over Time 
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Notes: All comparisons use a two-sided hypothesis test. Statistical significance levels for two-sided tests are indicated with asterisks, as 
follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


Sample Sizes and Sources: Treatment: 8,672. Control: 5,044. National Directory of New Hires. 


Exhibit 4.3: Impacts on Quarterly Employment Over Time 
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Notes: All comparisons use a two-sided hypothesis test. Statistical significance levels for two-sided tests are indicated with asterisks, as 
follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Sample Sizes and Sources: Treatment: 8,672. Control: 5,044. National Directory of New Hires. 
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4.2 Impacts on Other Labor Market Outcomes 
e HPOG increased employment in healthcare. 


According to the three-year survey, HPOG increased the rate of employment in the healthcare 
sector: 56 percent of the treatment group was employed in healthcare, compared to 44 percent 
of the control group (Exhibit 4.4). This increase of 12 percentage points represents a 27 percent 
relative increase. The effect on healthcare employment was much larger than the effect on 
overall employment (see Exhibit 4.1).22 


Exhibit 4.4: Impacts on Other Labor Market Outcomes 


Level of Treatment Control Relative 
Outcome Evidence Group Mean Group Mean Impact Impact 
Current or most recent job is in Secondary 55.6 43.9 11.6### 26.5 
healthcare 
Current or most recent job offers Secondary 58.5 56.2 2.3## 4.1 
health insurance 
Self-reported favorable job Exploratory 30.7 28.2 2.5 8.9 
characteristics 


Notes: Confirmatory and secondary findings use a one-sided hypothesis test, and exploratory findings use a two-sided hypothesis test. Self- 
reported favorable job characteristics reflects agreement with four favorable elements of job characteristics (flexibility, supportiveness, benefits, 
and advancement). See Appendix D for more information. 

Statistical significance levels for one-sided tests are indicated with hashtags, as follows: ##* = 1 percent; ## = 5 percent; * = 10 percent. 
Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Sample Sizes and Sources: 

Treatment: 6,027. Control: 3,305. Three-year survey. 


e HPOG increased access to employer-provided health insurance and a supportive 
work environment. 


Non-wage characteristics of employment reflect a key aspect of compensation that is often 
lacking in jobs for low-wage workers. HPOG led to slight improvements in several non-wage job 
characteristics (See Exhibit 4.4). HPOG increased the proportion of individuals whose current or 
most recent job offers health insurance from 56 percent to 58 percent; this statistically 
significant difference of 2 percentage points represents a 4 percent relative increase. 


In addition, 41 percent of the treatment group reported having a supportive working 
environment, compared to 38 percent of the treatment group (see Exhibit 4.4).*3 This difference 
of 3 percentage points represents a relative increase of 9 percent. 


e HPOG increased career progress, defined as completion of training and earnings 
growth. 


Educational progress and earnings growth reflect improvement in both confirmatory outcome 
domains, which in combination is one way to define career progress. Exhibit 4.5 reports impacts 
on this measure of career progress. We find that 43 percent of the treatment group completed 


22 This is true both for employment measured using NDNH and for employment as reported in the survey. See 
Appendix D for more information. 

23 A job is considered to have a supportive working environment if the worker reports having access to a combination 
of family-friendly policies, helpful coworkers and supervisors, high job satisfaction, fringe benefits, and opportunities 
for advancement. See Appendix B for more information on this measure. 
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training and experienced earnings growth, compared to 36 percent of the control group. This 
impact of 7 percentage points represents a 19 percent relative increase. 


When we examine the components of this measure we find large impacts for training completion 
only. As a result, we conclude that the career progress impact is being driven by higher training 
completion within the treatment group; participants in the treatment and control groups 
experienced similar levels of earnings growth. In Chapter 8 we describe the patterns of training 
and employment experiences for those individuals in the treatment group. 


Exhibit 4.5: Overall Impacts on Career Progress 


Level of Treatment Control Relative 
Outcome Evidence Group Mean Group Mean Impact Impact 
Training completion and earnings Secondary 43.0 36.2 6.8### 18.9 
growth (%)@ 
Training completion (%)» Exploratory 75.2 62.7 12,5 19.9 
Earnings growth (%)¢ Exploratory 57.7 57.3 0.4 0.7 


Notes: Confirmatory and secondary findings use a one-sided hypothesis test, and exploratory findings use a two-sided hypothesis test. 
Statistical significance levels for one-sided tests are indicated with hashtags, as follows: *** = 1 percent; ## = 5 percent; * = 10 percent. 
Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Sample Sizes and Sources: 

@ Treatment: 6,027. Control: 3,305. National Directory of New Hires and three-year survey. 

> Treatment: 6,027. Control: 3,305. Three-year survey. 

© Treatment: 6,027. Control: 3,305. National Directory of New Hires. 


4.33 Summary 


The HPOG logic model suggests that the training and services provided by HPOG should 
produce earnings gains for participants. However, despite the increase in training completion 
reported in Chapter 3, in this chapter we found no evidence of earnings impacts in the three 
years after random assignment. Although both treatment and control group members saw 
increases in their levels of earnings over time—trising from about $2,000 in the quarter of 
random assignment to $5,000 three years later—treatment group outcomes were no better than 
the control group. The high levels of employment over the three years of study follow-up in both 
the treatment and control groups (more than 80 percent received some employment earnings in 
either the 12th or 13th quarters after randomization) imply that increases in earnings need to be 
realized through higher wages or more hours, not just through entry into employment. 


Although there is no impact on earnings, a confirmatory outcome, the results in this chapter 
imply that HPOG caused some changes in other labor market outcomes. Notably, HROG 
produced a slight increase in overall employment and a sizeable increase in employment in 
healthcare (a 27 percent increase relative to the control group). Treatment group members also 
reported being employed in somewhat higher-quality jobs, as measured by having greater 
access to employer-provided health insurance and being more likely to report having a 
supportive work environment. 


The HPOG FOA noted that shifting employment to healthcare would imply greater earnings, but 
that does not seem to be the case as of the three-year follow-up. A plausible explanation for the 
findings in this chapter is that jobs in healthcare are perceived to be higher quality despite not 
necessarily offering higher wages (and perhaps offering lower wages). To assess this 
possibility, we examine specific trainings that study members received and the subsequent 
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occupations they reported in the follow-up survey (See Chapter 8). These descriptions confirm 
that the most commonly taken trainings and occupations varied substantially in their expected 
earnings. Two tracks emerge: short-term trainings generally lead to low-earnings occupations 
(e.g., Nursing Assistant), whereas longer-term trainings generally lead to better-paying 
occupations (e.g., Registered Nurse). This apparent split in the sample—as elaborated in 
Chapter 8—creates alternative experiences, which apparently serve to result in no detectable 
impact on earnings for the sample as a whole. 
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Summary of Key Findings: General Well-Being 


e HPOG did not decrease public assistance use overall. Forty-eight (48) percent of the treatment group 
received TANF, SNAP, or Medicaid in the month prior to the three-year survey, compared to 46 percent of 
the control group. Of the elements comprising this measure, receipt of TANF decreased and receipt of 
Medicaid increased for those in the treatment group. 


e HPOG had no detectable impact on personal or household income. This measure includes income 
from all sources, including job earnings and public assistance benefits. 


e HPOG reduced financial hardship. A smaller share of the treatment group reported generally not having 
enough money to make ends meet at the end of the month over the past year, compared to the control 
group. 

e HPOG improved psychological well-being and had no impact on barriers to school, work, job 
search, or family responsibilities. HPOG improved treatment group members’ core self-evaluation and 
reduced their perceived stress, compared to the control group. There was no detected difference in reports 
of childcare, transportation, or health conditions serving as a barrier. 


In this chapter, we report HPOG’s impact on general well-being. We examine outcomes across 
the domains of public assistance benefit receipt, economic status, financial hardship, 
psychological well-being, and barriers to school, work, job search, or family responsibilities. 
HPOG’s logic model suggests improvements in these domains might arise as a result of 
improvements to educational and labor market outcomes. Given that HPOG improved 
educational progress (Chapter 3) but not earnings (Chapter 4), expectations for impacts within 
the well-being domain are ambiguous: if there are well-being benefits to educational progress 
alone, then we might expect changes in this domain; but if well-being benefits arise only via 
labor market improvements, then expectations for well-being impacts would be limited to just 
those subgroups for which we observed labor market impacts. 


Abt Associates 
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Overview of Outcome Measures: General Well-Being 


The outcomes in this chapter are based on participant responses to the three-year survey. Two secondary measures 
are of particular interest: Individual receipt of TANF, SNAP, or Medicaid is a measure of public assistance receipt in 
the month prior to the three-year survey. Financial hardship is a self-reported measure of whether the participant 
reports typically not having enough to make ends meet at the end of the month over the past year. 


Outcome Outcome Type Variable Description | 
Public Assistance Benefits 

Individual receipt of TANF, Secondary Participant personally received benefits from TANF, SNAP, or Medicaid in the prior 
SNAP, or Medicaid month 

Individual receipt of TANF — Exploratory Participant personally received benefits from TANF in the prior month 

Individual receipt of SNAP Exploratory Participant personally received benefits from SNAP in the prior month 

Individual receipt of Exploratory Participant personally received benefits from Medicaid in the prior month 

Medicaid 

Economic Status 

Personal income Exploratory Total personal income received from all sources in prior month 

Household income Exploratory Total household income received from all sources in prior month 

Financial Hardship 

Participant reports generally Secondary Self-reported measure of whether the participant reports generally not having 

not having enough money enough to make ends meet at the end of the month over the past year 

to make ends meet at the 

end of the month 

Used loans to pay for Exploratory Used loans in either own name or parent's name to pay for school or living 

school or living expenses expenses since study enrollment 


Psychological Well-Being 


Core self-evaluation Exploratory A 12-item scale measuring an individual's evaluation of own abilities and control, 
scale ranges from 1 to 4: 


| am confident | get the success | deserve in life. 

Sometimes | feel depressed. 

When | try, | generally succeed. 

Sometimes when | fail | feel worthless. 

| complete tasks successfully. 

Sometimes, | do not feel in control of my work. 

Overall, | am satisfied with myself. 

| am filled with doubts about my competence. 

| determine what will happen in my life. 
10. |! donot feel in control of my success in my career. 
11. | am capable of coping with most of my problems. 
12. There are times when things look pretty bleak and hopeless to me. 


SMONOMNR WNP 


Perceived stress Exploratory A four-item scale measuring perceived stress, scale ranges from 0 to 4 

Barriers to School, Work, Job Search, or Family Responsibilities 

Childcare barrier Exploratory Indicator for whether childcare arrangements very often interfered with school, 
work, job search, or family responsibilities 

Transportation barrier Exploratory Indicator for whether transportation very often interfered with school, work, job 
search, or family responsibilities 

Illness or health barrier Exploratory Indicator for whether illness or another health issue very often interfered with 


school, work, job search, or family responsibilities 
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5.1 Impacts on Public Assistance, Income, and Financial Hardship 
e HPOG did not decrease public assistance use overall. 


We found that overall, 48 percent of the treatment group personally received benefits from one 
or more of TANF, SNAP, or Medicaid in the month prior to the three-year survey, compared to 
46 percent of the control group (Exhibit 5.1). When we focus on the elements that make up this 
measure, however, we found a decrease in TANF use—of 0.8 percentage points (5.6 percent of 
the treatment group versus 6.4 percent of the control group). We observed an increase in 
Medicaid use of 2 percentage points (37 percent of the treatment group versus 35 percent of the 
control group, for a relative increase of 5 percent).*“7° There was no detectable impact in 
receipt of SNAP (about 34 percent of both the treatment and control groups was receiving 
SNAP). 


Exhibit 5.1: Impacts on Public Assistance Benefits 


Level of Treatment Control Relative 
Outcome Evidence Group Mean Group Mean Impact Impact 
Individual receipt of TANF, SNAP, or Secondary 47.6 46.3 13 2.8 
Medicaid in prior month (%) 
Individual receipt of TANF (%) Exploratory 5.6 6.4 -0.8" -12.3 
Individual receipt of SNAP (%) Exploratory 33.9 33.8 0.1 0.2 
Individual receipt of Medicaid (%) Exploratory 37.1 35.2 1.9** 5.4 


Notes: Confirmatory and secondary findings use a one-sided hypothesis test, and exploratory findings use a two-sided hypothesis test. 
Statistical significance levels for one-sided tests are indicated with hashtags, as follows: #** = 1 percent; ## = 5 percent; * = 10 percent. 
Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Sample Sizes and Sources: 

Treatment: 6,476. Control: 3,551. Three-year survey. 


e HPOG did not have a detectable impact on personal or household income. 


Exhibit 5.2 shows that, as of the three-year survey, there was no detectable difference between 
the treatment and control groups in personal or household income. These measures of income 
captured income from all sources, including job earnings and private and public assistance. 


24 The increase in Medicaid receipt has several potential explanations. First, state expansions of Medicaid over 2014- 
2016 coincided with this study's follow-up; the support services offered to those in HPOG may have encouraged 
application to Medicaid and/or removed some of the institutional barriers that might have inhibited application. 
Another possibility is that treatment group members had more access to Medicaid via a stronger connection to 
healthcare training and employment. This could reduce the administrative and informational barriers associated with 
Medicaid enrollment or employers could ensure that people get signed up. Still another possibility is that treatment 
group members may have had a greater understanding of the importance of health insurance (an artifact of the 
impact on employment in healthcare) and made sure they were covered in any way they could (employer-sponsored 
or Medicaid). 


25 Medicaid receipt is available for HPOG-only grantees at baseline and in the short-term. For that group, levels of 
Medicaid receipt evolved as follows over time: about 45 percent of both the treatment and control group were 
receiving Medicaid at baseline; about 46 percent of both the treatment and control group were receiving Medicaid at 
the short-term follow-up; and about 39 percent of the treatment group and 38 percent of the control group were 
receiving Medicaid at the three-year follow-up. 
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Exhibit 5.2: Impacts on Income and Financial Hardship 


Level of Treatment Control Relative 
Outcome Evidence Group Mean Group Mean Impact Impact 
Personal income received from all Exploratory 1,668 1,668 -1 -0.0 
sources in the prior month ($) 
Household income received from all Exploratory 2,952 2,985 -32 -1.3 
sources in the prior month ($) 
Participant reports generally not Secondary 24.1 27.3 —3.2H0# -11.6 


having enough money to make ends 

meet at the end of the month (%) 

Used loans in either own name or Exploratory 24.8 25.5 -0.6 =2:9 
parent's name to pay for school or living 

expenses since study enrollment (%) 


Notes: Confirmatory and secondary findings use a one-sided hypothesis test, and exploratory findings use a two-sided hypothesis test. 
Statistical significance levels for one-sided tests are indicated with hashtags, as follows: #** = 1 percent; ## = 5 percent; * = 10 percent. 
Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Sample Sizes and Sources: 

Treatment: 6,476. Control: 3,551. Three-year survey. 


e HPOG reduced self-reported financial hardship. 


As of the three-year survey, 24 percent of the treatment group reported generally not having 
enough money to make ends meet at the end of the month over the past year, compared to 27 
percent of the control group (see Exhibit 5.2). This decrease of 3 percentage points represents 
a 12 percent relative improvement. A possible explanation for this impact is that the increased 
healthcare coverage (we observed impacts on both employer-provided insurance and Medicaid 
use) could be providing a cushion for treatment group members that results in their higher self- 
reported ability to make ends meet. 


5.2 Impacts on Psychological Well-Being and Barriers 
e HPOG slightly improved psychological well-being. 


HPOG led to slight improvements in two measures of self-reported psychological well-being. On 
a measure of core self-evaluation, HPOG led to a slight increase of 0.05 points on a scale from 
1 to 4 (Exhibit 5.3). The levels of this measure for both the treatment and control group imply 
more favorable psychological well-being. The interpretation of this measure changes between 
values of 2 and 3, where responses go from “disagree” to “agree.”*° Similarly, on a measure of 
perceived stress, HPOG led to a decrease of 0.05 points on a scale from 0 to 4, for a relative 
decrease of 2 percent. Both of the improvements to psychological well-being are statistically 
significant but relatively small. 


26 We reverse code negative measures so that all items in the scale reflect favorable well-being. 
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Exhibit 5.3: Impacts on Psychological Well-Being and Barriers 


Level of Treatment Control Relative 
Outcome Evidence Group Mean Group Mean Impact Impact 


Psychological Well-Being 


Core self-evaluation (range is 1 to 4) Exploratory : 3.23 0.05*** 15 


Perceived stress (range is 0 to 4) Exploratory 

Barriers to School, Work, Job Search, or Family Responsibilities 

Childcare arrangements (%) Exploratory : . ; 
Transportation (%) Exploratory 10.8 10.9 -0.0 -0.4 
An illness or health condition (%) Exploratory 9.0 8.9 0.1 1.2 


Notes: Confirmatory and secondary findings use a one-sided hypothesis test, and exploratory findings use a two-sided hypothesis test. 
Statistical significance levels for one-sided tests are indicated with hashtags, as follows: #** = 1 percent; ## = 5 percent; * = 10 percent. 
Statistical significance levels for two-sided tests are indicated with asterisks, as follows: ** = 1 percent; ** = 5 percent; * = 10 percent. 
Sample Sizes and Sources: 

Treatment: 6,476. Control: 3,551. Three-year survey. 


e HPOG did not have a detectable impact on self-reported barriers to school, work, job 
search, or family responsibilities. 


In the three-year survey, individuals were asked whether certain barriers “fairly often” or “very 
often” interfered with school, work, job search, or family responsibilities.2’ Both the treatment 
and control group reported few barriers: about 15 percent reported that childcare arrangements 
were a barrier; 11 percent reported that transportation was a barrier; and 9 percent reported that 
an illness or health condition was a barrier (See Exhibit 5.3). 


5.3. Summary 


The HPOG logic model posits that the training and services provided by HPOG will lead to 
improved educational and labor market outcomes, which in turn will lead to improvements in 
economic and general well-being. As noted in the introduction to this chapter, because HPOG 
did not produce impacts on earnings after three years, we might not expect an impact on 
economic and general well-being. Nevertheless, we observe that HPOG generally had modest 
impacts on a variety of measures of well-being. HPOG did not reduce overall public assistance 
use. 


Although HPOG did not increase individual or household income, HPOG did reduce reports of 
financial hardship. This result may stem from the increase in access to employer-provided 
health insurance (Chapter 4), the increased Medicaid receipt, the reduction in TANF use, or 
other factors associated with these changes. 


Finally, HPOG led to slight improvements in self-reported psychological well-being. Both the 
treatment and control group generally reported high assessments of their own abilities and 
moderate levels of stress. HPOG had no impact on barriers to school, work, job search, or 
family responsibilities, suggesting that other factors are likely driving the improvements in 
psychological well-being—such as satisfaction from completing training or less stress due to 
lower levels of financial hardship. 


27 Note the phrasing of this question differed from baseline materials that asked about barriers to school or work only. 
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Summary of Key Findings: Subgroups 
For demographic subgroups 


e HPOG increased educational progress more for those with dependent children; there were no 
differential earnings impacts. Across subgroups defined by participants’ race/ethnicity, age, or whether 
they had dependent children, most of the impacts of HPOG reflected the overall estimated impact on these 
outcomes. 


For policy-relevant subgroups 


e HPOG increased educational progress for subgroups defined by baseline enrollment in training, 
educational attainment, barriers to school/work, and employment. Those who came into the program 
seemingly less prepared to engage in a program such as HPOG fared better in educational progress than 
their counterparts who were seemingly more prepared. 


e There were differential impacts on earnings for subgroups defined by educational attainment and on 
employment for subgroups defined by barriers to school/work. Those who came into the program 
seemingly more prepared to engage in a program such as HPOG fared better on these measures than their 
counterparts who were less prepared. 


For baseline public assistance status-defined subgroups 


e HPOG increased educational progress more for those who were receiving WIC or SNAP at baseline, 
compared to those who were not receiving public assistance at baseline. Treatment group members who 
received WIC or SNAP at baseline saw a 16 percentage point increase in training completion relative to 
their control group counterparts. There were no differential earnings impacts. 


This chapter reports variation in impacts for selected subgroups of interest. We define our 
subgroups in three categories: (1) baseline measures of demographic characteristics 
(race/ethnicity, age, and presence of dependent children); (2) what we call “policy-relevant” 
characteristics; and (3) receipt of public assistance.” For which outcomes we report subgroup 
results varies with the particular subgroups, as pre-specified in the Three-Year Analysis Plan 
(Litwok et al. 2018):?9 


e For the demographic subgroups, we report impacts on the confirmatory outcomes 
(educational progress and earnings). 


e For the policy-relevant subgroups—those defined by enrollment in training, educational 
attainment, barriers to school/work, and employment, all at baseline—we report impacts on 


28 For ease of interpretation, this chapter's policy-relevant subgroups are defined by two groups. Appendix F reports 
results for four education-defined subgroups and for three barriers-defined subgroups. Appendix F also reports 
impacts for the subgroups defined by expectations for participating in HPOG full-time or part-time. We do not report 
these subgroups results in this chapter because the association between expectations for participating in HPOG full- 
time (or part-time) and actual participation in HPOG (be it full-time or part-time) is very weak. As a consequence, the 
results do not shed light on the question of whether actually participating full- versus part-time has differential 
impacts. 


29 As a result, we do not include an Overview of Outcome Measures textbox in this chapter. All outcomes that appear 
in this chapter also appear—and have been defined—in prior chapters. 
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confirmatory and secondary outcomes. These subgroups are labelled “policy-relevant” 
because any differential impact could have implications for changes in program design, 
implementation, or policy. For instance, a program could choose to target people with more 
or less education if the program appears to have larger impacts for that group. 


e For the subgroup defined by baseline public assistance use (i.e., receiving TANF, SNAP or 
WIC), we report impacts on confirmatory and secondary outcomes, as well as on those 
exploratory outcomes related to public assistance status at follow-up three years after 
randomization. This subgroup directly identifies a key HPOG target population as reflected 
in the FOA: TANF recipients. 
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Understanding Subgroup Impacts 


Whereas prior chapters reported the overall impact of HPOG, this chapter focuses specifically on HPOG’s impacts for 
subgroups. Broadly, we use the term subgroup to mean a subset of study participants who share some individual 
characteristic at baseline, as defined in the table below. 


Subgroup Type _ Subgroup Comparisons 
Demographic Race/Ethnicity: Hispanic/Latino vs. Black, non-Hispanic vs. White, non-Hispanic 
Age: Less than 25 years vs. 25 years or older 


Dependent children: No dependent children vs. One or more dependent children 
Policy-Relevant Enrollmentin training: — Enrolled vs. Not enrolled 


Educational attainment: Some postsecondary or more vs. High school education or less 
Barriers to school/work: No barriers vs. One or more barriers 
Employment: Employed vs. Not employed 

Public Assistance Public assistance status: TANF vs. SNAP/WIC only vs. No assistance 


We estimate impacts within each subgroup—e.g., the impact for educational progress on those younger than age 25—as 
well as differences in impacts across subgroups—e.g., the difference in impacts for educational progress between those 
younger than age 25 and those older than age 25. 


This chapter focuses on estimates of between-subgroup differential impacts. Although we report within subgroup impacts in 
exhibits, we do not focus on discussing these results because slicing the sample into subgroups reduces the ability to detect 
Statistically significant impacts. This implies that, although the overall effect might be statistically significant, the impact might 
not be statistically significant within any one subgroup simply by virtue of the reduced sample size. Instead, we do focus on 
subgroup impacts when one subgroup’s impacts are statistically different from its complementary subgroup’s impacts. 


Consistent with this strategy, tables in this chapter include an additional column that reports statistical significance for two- 
sided tests of differences in impacts. These results appear in the right-most column of the tables, as in the example below. 
The sample exhibit shows an impact of 8 percentage points on educational progress for those already enrolled in training at 
baseline and an impact of 15 percentage points for those not already enrolled at baseline. The right-most column indicates 
that the difference in impacts of 7 percentage points is statistically significantly different from zero. Because that difference is 
flagged as statistically significant, our discussion notes that there are differences in impacts for this subgroup, although we 
do not discuss the within-subgroup-specific impacts. Note that all differential impacts are assessed at the 10 percent level 
and denoted by daggers (additional details on the subgroup estimates and associated statistics are available in Appendix F 
for interested readers). 


Sample Exhibit: Impacts on Educational Progress by School Enrollment at Baseline 


Treatment Control Treatment Control 
Group Group Group Group Difference 
Outcome Mean Mean Impact Mean Mean Impact in Impacts 
Enrolled Nola alae) (=e) 
'% of sample 


Completed training (%)@ 84.5 76.8 ith? 72.9 57.9 15.0** 


Notes: Statistical significance levels for two-sided tests are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Statistical significance levels for two-sided tests of differences in impacts across subgroups are indicated as follows: t = 10 percent. 
Sample Sizes and Sources: 

a Treatment: 6,027. Control: 3,305. Three-year survey. 
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Exhibit 6.1: Impacts on Educational Progress by Demographic Subgroup 


Hispanic/Latino Black, non-Hispanic White, non-Hispanic Hispanicvs. Blackvs. _ Hispanic vs. 


(24% of sample) (ee ET UL) (CAO ET AL) ECG White White 
Completed training (%) T15 57.1 14.4" 73.3 60.9 12.4*** 79.3 68.2 11.1 2.0 13 3.2 
Average quarterly earnings during the 12th 4,506 4,633  -127 4,908 4,749 159 5,445 5,408 37 -285 122 -163 
and 13th quarters ($) 

Less Than 25 Years 25 Years or Older Less Than 25 Years 
Age (32% of sample) (xO mT UL) vs. 25 Years or Older 
Completed training (%) 71.8 59.0 12.8" 76.6 64.3 12.3%" 0.5 
Average quarterly earnings during the 12th 4,750 4,710 39 5,162 5,123 39 1 
and 13th quarters ($) 
No Dependent Children One or More Dependent Children No Dependent Children vs. 

Dependent Children (37% of sample) (CMO ET L)) One or More Dependent Children 
Completed training (%) 74.6 64.2 10.4** 75.6 62.0 13.7" -3.3t 
Average quarterly earnings during the 12th 5,065 5,005 61 5,023 4,992 30 30 


and 13th quarters ($) 

Notes: Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Statistical significance levels for two-sided tests of differences in impacts across subgroups are indicated as follows: t = 10 percent. 

Sample Sizes and Sources: 

a Treatment: 6,027. Control: 3,305. Three-year survey. 

> Treatment: 8,672. Control: 5,044. National Directory of New Hires. 
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6.1 Demographic Subgroups 


e Educational progress and earnings impacts were mostly the same across 
demographic subgroups. However, HPOG increased educational progress relatively 
more for study participants who reported having dependent children at baseline. 


The overall, pooled impact on educational progress of 13 percentage points (see Exhibit 3.1) 
reflects the impacts that subgroups experienced. As shown in Exhibit 6.1, the analysis finds little 
or no evidence of between-subgroup differences in impacts three years after randomization. 
The exception is the subgroup defined by presence of dependent children, where those study 
participants who reported having dependent children at baseline experienced a larger impact on 
educational progress than those with no dependent children at baseline (14 percentage points 
versus 10 percentage points). 


The results indicate that HPOG is effective at increasing educational progress for a wide variety 
of people, regardless of their race/ethnicity, age, or parental status. Similarly, none of these 
demographic groups showed any greater or lesser impacts on earnings three years after 
randomization, also in line with the analysis’s overall estimated impact. 


6.2 Policy-Relevant Subgroups 


e Across all policy-relevant subgroups, HPOG’s educational progress impacts were 
larger for those who came to the program seemingly /ess prepared to participate ina 
program like HPOG. 


As shown in Exhibit 6.2, across all of the policy-relevant subgroups we examined, impacts on 
training completion were consistently larger for those who came into HPOG seemingly less 
prepared to participate in a program like HPOG. Those who were not already enrolled in 
training, had a high school education or less, had some barriers to school or work, or were not 
employed all experienced relatively larger impacts on training completion than did their more 
prepared counterparts. Differences in impacts varied across subgroups, from about 5 to 7 
percentage points. The impacts for the more-prepared subgroups are one half to two thirds the 
size of the impacts for the less-prepared subgroups. 
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Exhibit 6.2: Im 


pacts on Educational Prog ress b Selected Polic -Relevant Subgrou 


Enrolled = ~ Not Enrolled 


(27% of sample) (Evo mT) 
Training completion (%)@ 84.5 76.8 Loe 72.9 57.9 15.0** -7.3t 
Confidence in career knowledge (rangeis1 3.50 3.49 0.01 3.36 3.33 0.03* -0.02 
to 4)b 


Some Postsecondary or More High School Education or Less 


(55% of sample) (45% of sample) | | 


Training completion (%)° 82.3 71.9 10.4** 66.6 51.3 15.3" -4.8t 
Confidence in career knowledge (rangeis1 3.45 3.42 0.03** 3.34 3.32 0.02 0.01 
to 4)4 
No Barriers One or More Barriers | | 

(ORTH) EOS) 
Training completion (%)° 75.6 64.6 11.0" 14.7 59.0 15.6"* -4.6t 
Confidence in career knowledge (rangeis1 3.42 3.41 0.01 3.36 3.31 0.05** -0.04 
to 4)4 

Employed Not Employed 

(43% of sample) (57% of sample) | | 
Training completion (%)< 77.3 68.3 9.0"* 74.0 58.6 15.4*** -6.4t 
Confidence in career knowledge (rangeis1 3.42 3.41 0.02 3.39 3.35 0.04** = -0.02 
to 4)4 


Notes: Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Statistical significance levels for two-sided tests of differences in impacts across subgroups are indicated as follows: t = 10 percent. 

Sample Sizes and Sources: 

4 Treatment: 4,894. Control: 2,238. Three-year survey (excluding HPOG/PACE programs). 

> Treatment: 5,264. Control: 2,425. Three-year survey (excluding HPOG/PACE programs). 

¢ Treatment: 6,027. Control: 3,305. Three-year survey. 

4 Treatment: 6,476. Control: 3,551. Three-year survey. 


e HPOG improved selected labor market outcomes more for those who were relatively 
more prepared at study entry for a program like HPOG—those who entered with at 
least some postsecondary education and those with no barriers to school or work. 


Unlike the finding on educational progress (that impacts were consistently larger for those 
participants who entered seemingly /ess prepared), for two labor market outcomes there is 
evidence of differential impacts in the other direction—larger impacts for those seemingly more 
prepared at baseline to participate in a program like HPOG, as shown in Exhibit 6.3. 


For those who entered with at least some postsecondary education, earnings impacts were 
$303 larger during the 12th and 13th quarters than for those who entered with a high school 
education or less. For those who entered with no barriers to school or work, employment 
impacts in the 12th or 13th quarter were more favorable, an improvement of 3 percentage points 
relative to those who entered with one or more barriers. There were no differences in impacts 
across subgroups for employment in healthcare or jobs offering health insurance. 
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Exhibit 6.3: Im 


pacts on Labor Market Outcomes by Selected Policy-Relevant Subgrou 


Enrolled x Not Enrolled 


(27% of sample) (EVO RET} 
Average quarterly earnings during the 12th 6,769 6,730 39 4,767 4,625 142 -103 
and 13th quarters ($)@ 
Employment in 12th or 13th quarter (%)@ 86.6 85.9 0.6 83.2 81.1 2.1 =1.5 
Current or most recent job is in healthcare 67.8 57.5 10.3** 54.0 41.1 12.9" -2.7 
(%)? 
Current or most recent job offers health 68.0 65.4 2.5 58.9 57.0 19 0.6 
insurance (%)> 
Training completion and earnings growth (%)< 56.2 52.0 4.2 39.1 31.4 Lae -3.5 
Some Postsecondary or More ee Ge 
(55% of sample) (45% of sample) 
Average quarterly earnings during the 12th + 5,685 5,505 180* 4,287 4,410 -123 303t 
and 13th quarters ($)¢ 
Employment in 12th or 13th quarter (%)¢ 82.0 80.9 11 81.9 81.1 0.8 0.3 
Current or most recent job is in healthcare 58.7 46.0 120 51.6 41.2 10.4** 2.3 
(%) 
Current or most recent job offers health 59.2 56.7 2.5* 57.7 55.5 2.1 0.4 
insurance (%)e 
Training completion and earnings growth (%)f 47.6 41.5 6.1°* 37.3 29.3 8.0"* =1,9 
(65% of sample) (EOI) 
Average quarterly earnings during the 12th = 5,303 5,188 115 4,532 4,659 -127 242 
and 13th quarters ($)¢ 
Employment in 12th or 13th quarter (%)¢ 83.5 81.6 1.9 78.8 79.8 -1.0 2.9t 
Current or most recent job is in healthcare 56.0 44.2 17 54.7 43.1 11,5" 0.2 
(%)= 
Current or most recent job offers health 59.6 57.7 2.0 56.3 53.2 3.2* -1.2 
insurance (%)e 


Training completion and earnings growth (%)f 43.7 37.4 6.3" 41.6 33.6 8.0"* -17 


(43% of sample) (57% of sample) 
Average quarterly earnings during the 12th 6,079 5,953 125 4,304 4,315 =14 136 
and 13th quarters ($)¢ 
Employment in 12th or 13th quarter (%)¢ 89.0 88.4 0.6 77.0 75.7 13 -0.7 
Current or most recent job is in healthcare 64.2 52.7 1153 49.3 37.3 12.0" -0.5 
(%) 
Current or most recent job offers health 66.0 62.9 3.1* 53.0 51.1 19 1.2 
insurance (%)e 
Training completion and earnings growth (%)' 46.6 41.1 5.5 40.3 32.3 8.0"* -2.5 


Notes: Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Statistical significance levels for two-sided tests of differences in impacts across subgroups are indicated as follows: t = 10 percent. 

Sample Sizes and Sources: 

a Treatment: 7,115. Control: 3,501. National Directory of New Hires (excluding HPOG/PACE programs). 

> Treatment: 4,894. Control: 2,238. Three-year survey (excluding HPOG/PACE programs). 

© Treatment: 4,894. Control: 2,238. National Directory of New Hires and three-year survey (excluding HPOG/PACE programs). 

4 Treatment: 8,672. Control: 5,044. National Directory of New Hires. 

© Treatment: 6,027. Control: 3,305. Three-year survey. 

‘ Treatment: 6,027. Control: 3,305. National Directory of New Hires and three-year survey. 
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e HPOG increased individual receipt of public assistance more for those who were 
enrolled in training at baseline and those who entered with a high school education or 
less. 


We found differential subgroup impacts on public assistance receipt for two of the four policy- 
relevant category subgroups: those defined by enrollment in training at baseline and by 
education at baseline (Exhibit 6.4). Those who were enrolled at baseline had impacts on receipt 
of public assistance that were 6 percentage points larger than did those who were not enrolled. 
Those who came into the study with a high school education or less had impacts on receipt of 
public assistance that were 4 percentage points larger than those who entered with some 
postsecondary education or more. 


Chapter 5 discussed the extent to which impacts on receipt of public assistance were driven by 
receipt of Medicaid. Exhibit 6.4 shows that both Medicaid and SNAP, and not TANF, seem to be 
the drivers of these differential subgroup impacts on public assistance receipt. 


Exhibit 6.4: Impacts on General Well-Bei g b Selected Policy-Relevant Subgrou 


eel (ze) Not Enrolled : 


(27% of sample) (EcoRI) 
Individual receipt of TANF, SNAP, or 35.0 29.4 5.6"* 52.4 52.7 -0.4 6.0T 
Medicaid in prior month (%)a 
Individual receipt of TANF (%)@ 3.1 2.8 0.2 6.6 El -1.1 =13 
Individual receipt of SNAP (%)2 22.2 17.4 4.8** 37.5 39.2 -1.7 6.5t 
Individual receipt of Medicaid (%)@ 29.7 25.0 4.6** 43.0 42.6 0.4 4.2 
Participant reports generally not having 18.1 20.6 -2.5 26.4 29.4 -3.0** 0.5 


enough money to make ends meet at the 
end of the month (%)@ 
Some Postsecondary or More High School Education or Less | | 


(55% of sample) (45% of sample) 
Individual receipt of TANF, SNAP, or 41.2 41.7 -0.5 55.5 52.0 3.5" -4.0t 
Medicaid in prior month (%)» 
Individual receipt of TANF (%) 4.6 5.3 -0.8* 6.9 77 -0.8 0.0 
Individual receipt of SNAP (%)» 28.2 29.2 -1.0 40.9 39.5 13 -2.3t 
Individual receipt of Medicaid (%)> 32.2 31.7 0.4 43.2 39.6 3.6** -3.2T 
Participant reports generally not having 23.4 26.3 = 2,90 25.0 28.5 —3.5"% 0.6 


enough money to make ends meet at the 
end of the month (%)6 


No Barriers One or More Barriers Pe 
(GORY) EO TT) 

Individual receipt of TANF, SNAP, or 43.8 43.1 0.6 54.8 51.9 3.0* -2.4 
Medicaid in prior month (%)» 

Individual receipt of TANF (%)> 5.2 5.8 -0.6 6.4 74 =1.4 0.5 

Individual receipt of SNAP (%)» 30.0 30.1 -0.1 41.1 40.3 0.8 -0.9 

Individual receipt of Medicaid (%)> 34.6 33.0 16 41.8 39.0 2.8* -12 
Participant reports generally not having 22.1 25.1 =3,1% 27.9 31.0 3,1" 0.0 


enough money to make ends meet at the 
end of the month (%)b 
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Treatment Control Treatment Control 
Group Group Group Group Difference 
Outcome Mean Mean Impact Mean Mean Impact _inImpacts 
Employed Not Employed 
(43% of sample) (57% of sample) 

Individual receipt of TANF, SNAP, or 39.7 39.4 0.3 53.5 51.5 2.1 -1.7 
Medicaid in prior month (%)» 

Individual receipt of TANF (%)> 3.3 5.0 -1.7" 74 7.5 -0.1 -16 

Individual receipt of SNAP (%)> 27.5 26.5 11 38.6 39.3 -0.7 18 

Individual receipt of Medicaid (%)» 30.4 30.4 -0.0 42.3 38.9 30%" 3.5T 
Participant reports generally not having 20.5 22.6 =2.2% 26.8 30.7 =3:9"* 18 


enough money to make ends meet at the 
end of the month (%)» 


Notes: Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Statistical significance levels for two-sided tests of differences in impacts across subgroups are indicated as follows: t = 10 percent. 


Sample Sizes and Sources: 
a Treatment: 5,264. Control: 2,425. Three-year survey (excluding HPOG/PACE programs). 
> Treatment: 6,476. Control: 3,551. Three-year survey. 


6.3. Baseline Public Assistance Status-Defined Subgroups 


This final section of this chapter examines impacts three years after randomization for 
subgroups defined by their public assistance status at baseline. We use a three-way 
classification: (1) individuals who were receiving TANF; (2) those who were receiving SNAP or 
WIC but not TANF; and (3) those who were not receiving SNAP, WIC, or TANF (“no 
assistance”).°° Nearly all of the individuals receiving TANF at baseline were also receiving 
SNAP. Within the study sample, these three subgroups represent 12 percent (TANF), 46 
percent (SNAP/WIC only), and 42 percent (no assistance), respectively. 


e All three subgroups made educational progress; those receiving only SNAP/WIC at 
baseline made more progress than those receiving no public assistance at baseline. 


Exhibit 6.5 shows the results for the public assistance status-defined subgroups. The impact for 
the SNAP/WIC group was 7 percentage points greater than for the no assistance group. The 
differences in impacts between the TANF and SNAP/WIC groups and the TANF and no 
assistance groups were not distinguishable from zero. 


e There were no differences in labor market outcomes across subgroups defined by 
public assistance status at baseline. 


There were no detectable between-group differences for any of the labor market outcomes for 
any of the three groups. The findings for subgroups defined by public assistance status at 
baseline are consistent with the overall findings from the preceding chapters—a small increase 
in employment, larger increases in employment in healthcare, and no impact on earnings. This 
implies that HPOG is no more or less effective for any one of these groups in terms of their 
labor market outcomes. 


30 These group classifications are defined based on participation in TANF, SNAP, or WIC at baseline. Participants, 
including those in the “no assistance” subgroup, might be receiving other public benefits such as Medicaid, housing 
aid, or Unemployment Insurance. 
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e HPOG’s impacts on receipt of TANF, SNAP, or Medicaid varied for subgroups defined 
by baseline public assistance status. 


As shown in Exhibit 6.5, treatment group members who received only SNAP/WIC at baseline 
experienced a reduction of 2 percentage points in personal receipt of TANF at three years after 
randomization. This impact differed from those who did not receive any assistance at baseline. 
Receipt of SNAP was also lower for those who received only SNAP/WIC at baseline, as 
compared to either those receiving TANF at baseline or those receiving no assistance at 
baseline. 


HPOG increased receipt of public assistance, specifically Medicaid, for those individuals who 
were receiving TANF at baseline. This impact differed from the impact for those who were 
receiving only SNAP/WIC at baseline or those not receiving any assistance at baseline. 
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Exhibit 6.5: Impacts by Public Assistance Status-Defined Subgroups 


TANF SNAP/WIC No Assistance TANF vs. | TANF vs. SNAP/WIC 


Re ed eco es (12% of sample) (48% of sample) (41% of sample) None |SNAP/WIC vs. None 
Training completion (%)@ 73.1 60.1 13.0"* 74.1 58.4 1b." 77.8 69.2 8.6" 44 -2.7 7.1t 
Confidence in career knowledge (range is 1 3.39 3.36 0.03 3.40 3.36 0.04** 3.41 3.39 0.02 0.01 -0.01 0.02 
to 4) 
Peers TANF SNAP/WIC No Assistance TANF vs. 
(12% of sample) (48% of sample) (41% of sample) None |SNAP/WIC vs. None 
Average quarterly earnings during the 12th 3,856 3,969 -113 4,538 4,548 -11 5,980 5,830 150 -264 -102 -161 
and 13th quarters ($)¢ 
Employment in 12th or 13th quarter (%)¢ 79.6 77.2 24 79.7 80.0 -0.3 85.3 83.3 2.0* 0.4 27 -2.3 
Current or most recent job is in healthcare 51.6 40.2 11.4" 54.2 41.2 13.0" 58.5 48.6 9.9*** 15 -16 3.2 
(%)* 
Current or most recent job offers health 53.2 51.8 14 56.7 55.2 15 62.0 58.9 3.1* =17 -0.2 -16 
insurance (%)@ 
Training completion and earnings growth 39.7 34.2 5.5% 41.1 32.5 8.6" 46.5 41.1 5.4 0.0 -3.1 3.2 
(%)* 
Sraearn cen: TANF SNAP/WIC No Assistance TANF vs. 
(12% of sample) (48% of sample) (41% of sample) None |SNAP/WIC vs. None 
Individual receipt of TANF, SNAP, or 76.0 70.5 55°" 57.1 57.7 -0.5 26.4 24.5 1.9 3.6 6.0t -2.4 
Medicaid in prior month (%)° 
Individual receipt of TANF (%)° 17.8 17.9 -0.2 5.2 6.9 la 2.0 2.1 -0.1 -0.1 15 -1.6T 
Individual receipt of SNAP (%)° 62.5 59.0 3.5 42.8 44.6 -18 13.4 12.3 11 2.5 5.4t -2.9t 
Individual receipt of Medicaid (%)° 59.9 51.3 8.6*** 43.9 43.1 0.8 21.4 20.3 11 7.5t 7.8t -0.3 
Participant reports generally not having 33.5 38.8 —5.3"* 27.4 31.4 “A AMS 16.9 18.4 -1.6 -3.7 -1.2 -2.5 
enough money to make ends meet at the 
end of the month (%)° 


Notes: Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Statistical significance levels for two-sided tests of differences in impacts across subgroups are indicated as follows: t = 10 percent. 

Sample Sizes and Sources: 

a Treatment: 6,027. Control: 3,305. Three-year survey. 

> Treatment: 6,476. Control: 3,551. Three-year survey. 

© Treatment: 8,672. Control: 5,044. National Directory of New Hires. 

4 Treatment: 6,027. Control: 3,305. National Directory of New Hires and three-year survey. 
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6.4 Summary 


This chapter has examined impact variation on confirmatory and secondary outcomes for 
subgroups in three categories: demographic, policy-relevant, and public assistance status- 
defined. There is no clear pattern regarding which subgroups fare relatively better or worse. 
That is, both more and less prepared subgroups fare relatively better in some circumstances, on 
some outcomes. For an alternative perspective, summarizing the evidence of impact variation 
for a given outcome across the three categories reveals some patterns of policy interest. 


6.4.1 Educational Progress Subgroup Differential Impacts 


For all policy-relevant subgroups, HPOG had a larger impact on training completion for the 
subgroups that might be considered to be less prepared for a program like HPOG at baseline. 
For example, those with some postsecondary education or more might be perceived as being 
more prepared to engage in additional postsecondary training relative to those with a high 
school education or less, who might require additional, remedial preparation in order to take 
advantage of HPOG’s training offerings. Despite this characterization, in terms of educational 
progress, it is the less-prepared subgroups that appear to benefit more from HPOG than they 
would have otherwise, relative to their control group counterparts. Impacts on training 
completion were also larger for those who entered HPOG receiving SNAP/WIC versus those 
who came into the program receiving no public assistance. 


These findings about differential educational progress impacts—that they favor some less 
prepared subgroups—might arise for programmatic reasons. That is, it could be something that 
the local HPOG programs did to ensure that those with less education, for example, could 
meaningfully benefit. These findings might also arise as a function of the control group’s 
conditions and experiences. That is, in the absence of HPOG, control subgroups might have 
particularly poorer outcomes. The combination of these two forces might explain the differential 
educational progress impacts: the local HPOG programs’ offerings might facilitate 
disproportionate improvements within the less-prepared portion of its target population, and 
absent treatment, the control subgroups’ outcomes might be poorer. 


6.4.2 Labor Market Subgroup Differential Impacts 


Three years after random assignment, we also observe differential impacts for some subgroups 
in their employment and earnings. Concerning labor market outcomes, there is some evidence 
that those who enrolled in training with relatively greater education and without barriers to 
school or work fared relatively better. Those with more education had greater earnings impacts; 
and those with no barriers had greater employment impacts. 


6.4.3 Additional Exploration of Subgroup Differential Impacts 


To further explore these findings, we first analyzed subgroup findings for the four measures that 
make up training completion. Then, given those results, we examined the duration of trainings 
for selected subgroups (see Appendix F). The rationale for these analyses stems from 
heterogeneity in the HPOG sample. Some treatment group members enroll in short-term 
trainings, and it is our conjecture that those trainings are associated with lower-rung, lower- 
quality jobs, even if in the healthcare sector. In comparison, other treatment group members 
enroll in longer-term trainings, which more likely lead to better healthcare sector jobs. If these 
are two dominant paths within HPOG, then they might create offsetting earnings impacts: some 
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participants might fare worse and others might fare better as a result of their HPOG 
experiences. If these two effects are offsetting, that might explain the evaluation’s net impact of 
only $42 (not distinguishable from zero) in quarterly earnings after three years (see Exhibit 4.1), 
despite the larger share of treatment group members working in the healthcare sector. 


We used the education-defined subgroup to explore the extent to which there is evidence to 
support this conjecture. In brief, the analysis reveals differential impacts for completion of short- 
term training, but fewer differentials for completion of longer-term training as of the three-year 
follow-up. Of note, we observe an impact differential in completion of a certificate or degree 
requiring one or more years of college that favors the subgroup defined by greater baseline 
education—the same subgroup where we observe an earnings differential. This finding is 
consistent with our conjecture that those who complete longer-term training might find higher- 
wage jobs in the healthcare sector. We further explore this conjecture from a descriptive 
perspective in Chapter 8. 


The next chapter focuses on impacts within a particular subset of the sample—parents who had 
dependent children at baseline—and explores the extent to which there were impacts of HROG 
for outcomes related to child development and well-being. 
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Summary of Key Findings: Child Development and Well-Being 


e Impacts on the subset of the HPOG sample who were parents at baseline are similar to impacts for 
the full sample. HPOG improved outcomes in training completion, employment in healthcare, job quality, 
and career progress within the subset of the sample who were parents at baseline. 


e HPOG did not have detectable impacts on our measures of child development or well-being. We do 
not detect differences between the treatment and control groups in their children’s educational aspirations, 
socioemotional development, academic skills, or school-related risks. 


Programs such as HPOG may improve outcomes not only for participants, but also for their 
children. Given the differential educational progress impacts for participants with children 
(Exhibit 6.1), we focus here on whether those impacts translate to children. Impacts on children 
might arise in multiple ways. Added resources in the household may lead to improved child well- 
being. Children’s outcomes might improve from their having role models in their parents’ pursuit 
of more education and better work opportunities. However, if increasing parents’ work means 
that they are less able to supervise their children during afterschool hours, then it is possible 
that children’s well-being might decline. All three of these hypotheses—resources, role models, 
supervision—have been observed in prior research (Duncan, Morris, and Rodrigues 2011; 
Gennetian et al. 2002; Morris, Gennetian, and Duncan 2005). 


This chapter first discusses the impact of HPOG on parents and then turns to impacts on 
children’s outcomes. At baseline, 63 percent of the HPOG sample (treatment and control) had 
one or more dependent children; 84 percent were unmarried and 89 percent were female. 


7.1 Child Development and Well-Being Sample 


In this chapter, we investigate HPOG’s impact on children, focusing on preschool through high 
school age at follow-up. We focus on this age range because it excludes children who were 
born after random assignment, and it includes children who might plausibly be affected by their 
parents’ participation in HPOG during this follow-up period. We further restrict attention to 
parents who at baseline: 


e had legal custody of the child at least half-time; and 


e had provided basic identifying information for the child on the household roster collected 
through the study’s baseline survey. 


The parents examined in this analysis are those who had dependent children at baseline; and 
only the children who were part of the family at baseline are eligible to be the focus of the follow- 
up survey questions about children. Any children added to the sample members’ families after 
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random assignment are not part of this analysis, and the analysis does not consider changes to 
family composition that might be attributed to HPOG. ** 


For each parent of eligible children, the evaluation randomly selected a single focal child, and 
the questions in the follow-up survey’s child module asked specifically about this child. Asking 
questions about a specific child produces higher-quality data than asking about children 
generally. Although we could have duplicated the child module to gather data about multiple 
children, we ask the parents to answer for only one child, to limit the burden of the survey. 
Because we chose the focal child randomly for each household, the analyses are representative 
of all eligible children. *2 


Of the full baseline sample of 10,693 study participants in the HPOG-only programs, 5,966 were 
parents of one or more eligible children. The evaluation selected a single focal child for each of 
these parents. At the three-year follow-up, a total of 4,351 parents of focal children responded to 
the survey (73 percent of the parents of focal children) and were screened for the survey’s child 
module. Of these, 1,044 were ineligible to participate in the module: 708 parents did not have a 
child whose name and date of birth matched the selected focal child, 237 parents did not live 
with the child at least half-time, and 99 parents were ineligible for other reasons. Therefore, the 
survey collected data on 3,307 focal children. 


Because socioemotional and academic development vary by age and grade level, outcome 
measurement also varies across the following subsamples: preschool-age children, 
kindergarten through grade 5, and grades 6 through 12. Exhibit 7.1 gives the criteria used to 
define subsamples of parents who were asked tailored survey questions, the typical age range 
and sample sizes for the full child sample (top row), and the same for the age/grade-specific 
subsamples. 


As a result, the child development and well-being sample is much smaller than the overall 
sample, particularly for age/grade specific analyses. For example, there are only 750 preschool- 
age children. This reduction in sample size reduces statistical power, potentially limiting our 
ability to detect a true impact. 


31 The sample of parents in Chapter 7 differs from the sample of study participants with “one or more dependent 
children” in Chapter 6. In Chapter 6, the sample with one or more dependent children includes all study participants 
who reported having a child at baseline. In Chapter 7, the parent sample only includes parents who had at least one 
child selected to be the focus of follow-up survey questions. Most notably, the selection of focal children in 
HPOG/PACE programs was different than the selection in HPOG-only programs, so we excluded children from the 
HPOG/PACE programs from the survey sample (therefore, those parents were excluded from the parent subsample). 
We also excluded parents who did not provide sufficient identifying information for their children to ask the follow-up 
survey questions. 


82 We include weights in the analysis to adjust for the frame from which children were selected. This frame 
incorporates the number of children in the household. 
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Exhibit 7.1: Child Development and Well-Being Sample 
Typical Age Sample Sample Sample 


Range at Size: Size: Size: 

Sample Criteria Follow-Up Treatment Control Total 
Full child sample Expected to be of preschool age or enrolledin school  3to18 2,269 1,038 3,307 
(children of all (kindergarten to grade 12) at the three-year follow- 
grades/ages) up, where the expected date of high school 

graduation was calculated from the child’s date of 

birth collected at baseline 

In parent's legal custody at least half-time (at 

baseline and at time of three-year survey) 
Preschool age Not yet enrolled in kindergarten 3to5 521 229 750 
Grades K-5 Enrolled in kindergarten to grade 5 at follow-up 5to 11 1,009 463 1,472 
Grades 6-12 Enrolled in grade 6 to grade 12 at follow-up 11 to 18 718 327 1,045 


Notes: Typical Age Range at Follow-Up gives the 5th and 95th percentile of the age distribution, rounded to the nearest year. Surveys 
conducted in the summer asked about enrollment and school experiences in the previous school year. Age/grade-specific outcomes are 
unavailable for 40 children whose parents provided invalid information on enrollment grade. 


Overview of Outcome Measures: Child Development and Well-Being 


The outcomes in this chapter are based on parent response to the three-year survey. The three-year survey asked 
certain questions only of parents with children in particular grade ranges (those ranges are noted in the table 
below). All outcomes in this chapter are exploratory. 


Outcome Grade Range Variable Description 
Parent believes child All Indicator equals 1 if parent expects the child to do one of the following: 
will complete some e Complete some technical school after high school 
education beyond e Finish technical school after high school 
high school (%) e Complete some college 
e Finish college 
e Earn an advanced degree 


Indicator equals 0 if parent expects the child to do one of the following: 
e Complete some high school 


e Finish high school 


Child believes s/he All Indicator equals 1 if parent reports the child expects to do one of the following: 
will complete some e Complete some technical school after high school 

education beyond e Finish technical school after high school 

high school (%) e Complete some college 


e Finish college 

e Earn an advanced degree 

Indicator equals 0 if parent reports the child expects to do one of the following: 
e Complete some high school 

e Finish high school 
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Child development of _ preschool-5 Six-item scale measuring socioemotional skills, constructed by averaging across the 
socioemotional skills following items. Parent rates the child as far below average (-2), below average (-1), 
(range is -2 to 2) about average (0), above average (1), or far above average (2) on: 

e Ability to follow direction 

e Ability to make new friends 

e Ability to wait his/her turn in games or activities 

e Acting his/her age 

e Ability to concentrate or pay attention 

e Ability to control his/her temper 
Child development of preschool Three-item scale measuring academic skills, constructed by averaging across the 
academic skills only following items. Parent rates the child as far below average (-2), below average (-1), 
(range is -2 to 2) about average (0), above average (1), or far above average (2) on: 

e Ability to express themselves verbally 

e Early reading skills 

e Skills with math or numbers 


Child development of K-5 Two-item scale measuring academic skills, constructed by averaging across the 
academic skills following items. Parent describes teacher reports of the child doing not well at all 
(range is -2 to 2) (-2), below average (-1), about average (0), well (1) or very well (2) on: 

e Reading 

e Math 
Parental perception 6-12 Parent reports the child earned mostly A’s or mostly A’s and B's in school. If school 
of student does not give letter grades: parent reports the child was one of the best students in 
achievement (%) his/her class or above the middle in his/her class 
School-related risk K-12 Number of school-related risks (academic risk, attendance risk, and behavioral risk), 
(range is 0 to 3) ranging from 0 to 3. Scale reflects the number of domains where risk is present 
Academic risk (%) K-12 Indicates the parent reported at least one of the following: 


e Child has repeated at least one grade in school 
e Inthe current or most recent school year, a teacher has contacted an adult in the 


household about problems with the child's schoolwork 


Repeat grade (%) K-12 Parent reports the child has repeated at least one grade in school 
Schoolwork-Teacher K-12 In the current or most recent school year, parent reports a teacher has contacted an 
contact (%) adult in the household about problems with the child’s schoolwork 

Attendance risk (%) K-12 Indicates the parent reported at least one of the following: 


e Child was absent for more than 2 days in the last month 
e Child was late for school on more than 2 days in the last month 


Absent more than K-12 Parent reports the child was absent for more than 2 days in the last month 

two days last month 

(%) 

Late more than two K-12 Parent reports the child was late for school on more than 2 days in the last month 
days last month (%) 

Behavioral risk (%) K-12 Indicates the parent reported at least one of the following for the current or most 


recent school year: 
e A teacher has contacted an adult in the household about the child’s behavior 
problems in school 


e Child was suspended or expelled from school 


Behavior-Teacher K-12 In the current or most recent school year, parent reports a teacher has contacted an 
contact (%) adult in the household about the child’s behavior problems in school 

Suspend/expel in K-12 In the current or most recent school year, parent reports the child was suspended or 
current school year expelled from school 

(%) 


Notes: Additional information on the sample, sample sizes, and missing data handling appears in Appendix B.: 
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7.2 Impacts on Parents 


The logic model suggests that HPOG’s impacts on parents might yield impacts on their children. 
Following this chain of logic, we begin by reporting HPOG’s impacts on parents. This analysis 
only includes parents of focal children. 


e Impacts on parents are similar to impacts for the full sample. 


Exhibit 7.2 presents findings on HPOG’s impacts on confirmatory and secondary outcomes for 
parents of focal children.2? HPOG increased parents’ completion of training but yielded no 
detectable impact on their average earnings in the 12th-13th quarter after random assignment. 
Neither was there a detectable impact on their employment; however, HPOG increased 
employment in healthcare and access to employer-sponsored health insurance for parents. 


Exhibit 7.2: Impacts for Child Development and Well-Being Sample 


Treatment Control Relative 

Outcome Group Mean Group Mean Impact Impact 
Training completion (%)@ 77.2 61.9 15.4" 24.8 
Confidence in career knowledge (range is 1 to 4)> 3.42 3.38 0.03 0.93 
Average quarterly earnings during the 12th and 5,163 5,157 6 0.1 
13th quarters ($)° 
Employment in 12th or 13th quarter (%)¢ 85.4 84.1 12 14 
Current or most recent job is in healthcare (%) 61.5 46.8 14.8%" 31.6 
Current or most recent job offers health 63.9 60.6 3,3* 5.4 
insurance (%)@ 
Training completion and earnings growth (%)4 44.9 36.3 8.6"* 23.8 
Individual receipt of TANF, SNAP, or Medicaid in 59.4 58.0 1.4 2.3 
prior month (%)» 
Participant reports generally not having enough 26.3 27.8 -1.5 -5.3 


money to make ends meet at the end of the 
month (%) 


Notes: Career progress reflects a combination of training completion and earnings growth (see Chapter 4 for more information). Statistical 
significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Sample Sizes and Sources: 

@ Treatment: 2,080. Control: 951. Three-year survey. 

> Treatment: 2,248. Control: 1,019. Three-year survey. 

© Treatment: 2,248. Control: 1,019. National Directory of New Hires. 

4 Treatment: 2,080. Control: 951. National Directory of New Hires and three-year survey. 


The introduction to this chapter noted three plausible pathways for a program such as HPOG to 
influence children’s outcomes: household resources, the role modeling of parents, and parental 
supervision. Without overall impacts on earnings, impacts on child outcomes seem unlikely. 


33 Appendix G includes the full details of the subgroup analysis, comparing impacts for parents of focal children and 
impacts for other sample members. For several outcomes—confidence in career knowledge, employment, and 
financial hardship—there are impacts for the overall sample and not for the parents of focal children subsample. This 
is likely a function of the smaller sample size of the parent subsample decreasing the precision of estimates. The only 
detectable differential impacts between the parents of focal children and the non-parent subsample are for training 
completion and healthcare employment (see “Understanding Subgroup Impacts” textbox in Chapter 6). 
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However, because HPOG parents are in training to a greater degree than their control group 
counterparts, it is plausible that the role modeling and supervision mechanisms might influence 
children. The role modeling hypothesis would imply that parents’ dedication to education or 
training would improve children’s educational outcomes. However, if parental supervision is 
particularly important for adolescents, then parents’ absence due to additional education, 
training, or work could imply greater levels of delinquency among older children. 


The next section provides evidence on whether any of these mechanisms is strong enough to 
influence children’s development and well-being at the three-year follow-up. 


7.3. Impacts on Child Development and Well-Being 
e HPOG did not affect child development or well-being. 


We do not detect differences in educational aspirations, socioemotional development, academic 
skills, or school-related risk between the treatment and control groups (Exhibit 7.3). Across both 
the treatment and control group, parents report that their child is doing well. Nearly all parents in 
the full child sample (94 percent) expect their child will complete some education beyond high 
school. The same proportion of parents report that their child says he or she would like to 
complete some education beyond high school. 


Exhibit 7.3: Impacts on Child Development and Well-Being, for All Children and for 
Children by Grade Range 


Treatment Control Relative 
Outcome Sample Group Mean Group Mean Impact Impact 
Parent believes child will complete some Full child sample 94.1 94.2 -0.1 -0.1 
education beyond high school (%)@ 
Child believes s/he will complete some Full child sample 94.1 94.1 0.1 0.1 


education beyond high school (%)@ 


ocioemotional Development 


Child development of socioemotional Preschool -grade 5 0.45 0.43 0.02 41 
skills (range is -2 to 2)» 


Academic Skills 


Child development of academic skills Preschool age 0.43 0.48 -0.05 -11.0 
(range is -2 to 2)¢ 
Child development of academic skills Grades K-5 0.92 0.90 0.02 17 
(range is -2 to 2)4 
Parental perception of student Grades 6-12 72.1 76.2 -4.1 -5.4 


achievement (%)e 


chool Related Risk 


School-related risk (range is 0-3) Grades K-12 0.66 0.66 0.00 0.5 


Notes: Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Sample Sizes and Sources: 

@ Treatment: 2,269. Control: 1,038. Three-year survey. 

> Treatment: 1,530. Control: 692. Three-year survey. 

© Treatment: 521. Control: 229. Three-year survey. 

4 Treatment: 1,009. Control: 463. Three-year survey. 

© Treatment: 718. Control: 327. Three-year survey. 

‘ Treatment: 1,727. Control: 790. Three-year survey. 
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Exhibit 7.3 identifies specific samples from the child development and well-being sample in the 
second column. Across both treatment and control groups, parents perceive their preschool- 
grade 5 child to be slightly above average in socioemotional development and development of 
academic skills. 


Next, considering the whole K-12 grade range, the school-related risk score counts the domains 
where risk is present, with up to one point for each of the following: academic risk, attendance 
risk, and behavioral risk. The average school-related risk score of 0.66 in Exhibit 7.3 above 
indicates parents reported risks on average in fewer than one of these areas. 


Exhibit 7.4 below shows impacts on the measures and items that make up the school-related 
risk scale. Among those possible risk factors, parents most commonly reported teacher contact 
with concerns about the child’s schoolwork (20 and 21 percent for the treatment and control 
group, respectively) and/or about behavior (23 percent). The most severe risks were also the 
least commonly reported: repeated a grade (6 to 7 percent for the treatment and control group, 
respectively) or suspension or expulsion in current year (4 percent). There is no evidence of 
impact on any of the components of school-related risk. 


Exhibit 7.4: Impacts on School-Related Risk (Grades K-12 


Treatment Control Relative 

Outcome Group Mean Group Mean Impact Impact 
Academic risk (%) 24.6 24.8 -0.2 -0.8 
Repeat grade (%) 5.8 6.8 -0.9 -13.9 
Schoolwork-Teacher contact (%) 21.4 19.7 1.8 9.0 
Attendance risk (%) 17.1 17.3 -0.2 -1.0 
Absent more than two days last month (%) 11.4 11.4 0.0 0.1 
Late more than two days last month (%) 10.0 9.8 0.2 2.2 
Behavioral risk (%) 24.6 24.1 0.5 2.2 
Behavior-Teacher Contact (%) 23.2 22.9 0.3 1.3 
Suspend/expel in current school year (%) 3.7 3.5 0.2 5.8 


Notes: Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Sample Sizes and Sources: 
Treatment: 1,727. Control: 790. Three-year survey. 


7.4 Summary 


As discussed previously, the HPOG logic model posits that the training and services provided 
by HPOG will lead to improved educational and labor market outcomes, which in turn has the 
potential to influence family and child well-being. We analyze HPOG’s impacts on children to 
investigate the full extent of the logic model, but recognize that the path through which HPOG 
might influence children’s outcomes has many steps and many assumptions are embedded in 
the connections between each step. Given that we do not observe impacts on earnings or 
income—the “resources” mechanism through which HPOG might affect child outcomes—we 
might not expect to see impacts on children. However, the impacts on educational progress 
could yield “role model” related favorable impacts or “supervision” related unfavorable impacts 
among children. 


In fact, we do not observe any impacts on children, either favorable or unfavorable. Although the 
overall sample for the HPOG impact study is relatively large (13,716 individuals), the sample for 
impacts on children is smaller: 44 percent of the total sample was eligible for the child module 
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according to baseline information, and three-quarters of those who were eligible based on 
screening questions ultimately responded to the survey. As a result, the full sample for child 
outcomes is 3,307 individuals, 24 percent of the overall sample for the HPOG impact study. For 
outcomes measured for specific age groups, the resulting sample sizes are even smaller, 
ranging from 750 for outcomes measured for preschool age children to 2,517 for outcomes 
measured for all children enrolled in school (grades K to 12). Smaller sample sizes have lower 
statistical power, reducing the likelihood of detecting small impacts. Because we would not 
expect to see large impacts and are unlikely to detect small impacts, it is not surprising that we 
do not detect impacts on child well-being. 
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Summary of Key Findings: Patterns of Training and Employment 


e Treatment group members participated in training mostly in the early months after random 
assignment. The proportion of participants in training was highest in the first few months and declined 
steadily thereafter. 


e About 16 percent of treatment group survey-respondents enrolled in long-term training, defined as a 
year or longer. These individuals were more likely to have been enrolled in training or had some 
postsecondary education or more at baseline. 


e Three years after random assignment, 89 percent of those who enrolled in long-term training were 
employed, earning an average of $21.00 per hour. In contrast, 75 percent of those who enrolled in 
short-term training were employed, with an average wage of $14.50 per hour. More than two thirds of 
those who enrolled in long-term training and were employed at the three-year follow-up were working as 
either registered nurses, licensed practical nurses, or licensed vocational nurses; more than two thirds of 
those who enrolled in short-term training and were employed at the three-year follow-up were working as 
either Nursing, Psychiatric, or Home Health Aides or in another relatively low-wage occupation. 


e There is little evidence of participants returning for a second spell of training. According to the three- 
year survey, only nine percent of participants participated in training, became employed, and then began a 
second spell of training. This finding is the same for those who first enrolled in short-term and long-term 
training. 


The third research question for this three-year impact analysis is: To what extent do the 
education and employment experiences of HPOG participants over time suggest that they are 
following a career pathway? 


The career pathways framework originates in the observation that people can build careers by 
combining training and work, either simultaneously or sequentially, to progress up a career 
ladder. Combining training and work in such a manner seems particularly likely in the heavily 
regulated healthcare sector. The FOA states that successful local HPOG programs will “support 
participants’ advancement along a defined career pathway, such as an articulated career 
ladder, if such a pathway exists in the healthcare industry... [or develop] such pathways where 
they do not currently exist” (HHS/ACF/OFA 2010). 


To address this third research question, this chapter describes the patterns of training and 
employment for the HPOG treatment group over the three-year follow-up. The structure of the 
three-year survey—collecting a complete training, education, and employment history since 
study entry—allows us to identify these patterns within the survey’s three-year observation 
window. Using the survey’s history, we construct a month-by-month picture of when a survey 
respondent was (1) enrolled in training (whether in HPOG or elsewhere), (2) employed, (3) both 
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enrolled in training and employed, or (4) neither enrolled in training nor employed.™ In the 
remainder of this chapter, we refer to these four statuses (in italics) as training, employed, both, 
and neither, respectively. *° 


Throughout this analysis, we focus specifically on the comparison between study members 
whose first training after random assignment was long-term (we focus exclusively on training for 
Licensed and Vocational Nurse or Registered Nurse, 16 percent of the sample) versus short- 
term (all other trainings, 84 percent of the sample).*° Two findings noted in the preceding 
chapters imply that this comparison is particularly salient. First, in Chapter 4 we noted that it 
might be useful to think of two possible tracks of training and employment: some short-term 
trainings lead to low-earnings occupations (e.g., Nursing Assistant), whereas some longer-term 
trainings lead to better-paying occupations (e.g., Registered Nurse). Second, in Chapter 6 we 
found that selected subgroups—those with greater education and fewer barriers to school or 
work at baseline—did have favorable impacts on earnings. This chapter presents analyses that 
supplement these findings. 


The experiences we describe occurred after the point of random assignment into the HPOG 1.0 
Impact Study. As such, the experimental design implies that we could classify patterns as 
outcomes and compare these outcomes in the treatment group to the control group to estimate 
the impact of HPOG on career patterns. To do so would require us to designate certain patterns 
as indicative of “career progress” and others not. We hesitate to do so because career progress 
can have many possible definitions, manifest in many ways, and need not follow any pattern. 
Instead, we focus only on describing the patterns of treatment group members and remain 
agnostic about which patterns indicate career progress. Importantly, those patterns reflect a 
sample who began enrolling in training midway through the five-year HPOG grant period, 
leaving at most two and a half years during which the earliest enrollees could access the 
training that was HPOG grant-funded. 


This chapter proceeds as follows. We begin by describing the cumulative patterns of movement 
into training and employment for the treatment group. We then split the treated sample into two 
groups and focus specifically on patterns of training and employment for those who enrolled in 
short-term trainings versus those who enrolled in long-term trainings, which correspond to 
training for Licensed and Vocational Nurses or Registered Nurses and all other trainings, 
respectively. The observations that follow lead us to conclude that these two subsets of the 
treatment group have distinct HPOG experiences, in terms of their labor outcomes. Then we 
describe the most common patterns of training and employment and wage-earning potential for 
these two groups, considering how these experiences align with the underlying assumptions of 
the career pathways framework. We close with the implications of these findings for this report. 


34 Because the analysis of training and employment patterns relies on the three-year survey, we limit the discussion 
to survey respondents who did not skip out of the survey, which comprise 69 percent of the study sample’s treatment 
group (see Appendix B). 


35 Those in neither are not survey non-respondents; instead, they are respondents who did not report a single training 
or employment spell in the three years following random assignment. 


36 We identify these trainings using administrative data from the PRS. 
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8.1 Training and Employment Over Time 


Exhibit 8.1 shows the movement of the treatment group into training and employment over the 
three-year follow-up, with specific focus on the moves in the first six months. Consistent with the 
sample baseline characteristics described in Chapter 1, treatment group members enrolled in 
HPOG from various baseline statuses (Exhibit 1.2). According to the three-year survey, 
treatment group respondents recall that at the time of their random assignment: 


e 21 percent were training; 
e 24 percent were employed; 
e 17 percent were both; and 


e 38 percent were neither. 


These percentages correspond to the respondents’ status in month 0, on the left side of 
Exhibit 8.1.9” 


e Treatment group members participated in training mostly in the early months after 
random assignment. Levels of training and both returned to their baseline levels six 
and ten months after random assignment, respectively. 


Corresponding to initial enrollment in HPOG, the proportion of the treatment group in either the 
training or both status increased in the first few months following random assignment. After the 
first three months, the share of respondents in training or both began to decline, witha 
commensurate increase in the share of respondents in employed status. After a sharp initial 
decline, the share in neither remained relatively stable. 


In the next section we split the sample between those whose first training was not for Licensed 
and Vocational Nurses or registered nurses—which we refer to as short-term and those whose 
first training was for Licensed and Vocational Nurses or Registered Nurses—which we refer to 
as long-term. Werner et al. (2018) report that Licensed and Vocational Nurses or Registered 
Nurses are the only trainings with average duration among completers greater than 12 months. 
Within these two groups we assess the characteristics of employment three years after random 
assignment. 


37 The numbers reported in this chapter vary from data collected at the time of random assignment. According to the 
PRS and study's baseline survey, at random assignment, 43 percent of the sample reported being currently 
employed and 26 percent currently enrolled in school (See Chapter 1). The numbers here may vary for a variety of 
reasons, including: differences in the reference period (here we measure activities over a full month, rather than at 
the point of random assignment; differences in the sample (here we examine only treatment group survey 
respondents); recall bias; or some combination of these factors. 
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Exhibit 8.1: Training and Employment Status of Treatment Group Survey Respondents 
Over 36 Months 


Percentage of Sample (%) 
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Months Since Random Assignment 


ATraining @Employed ME Both @ Neither 


Notes: Each point represents the fraction of the treatment group survey respondents in the particular status (training, employed, both, neither) 
at each month after random assignment. 


Source: Three-year survey. 
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8.2 Patterns for Short-Term and Long-Term Training 


This section examines the portion of the treatment group that both enrolled in training according 
to the PRS and reported on the three-year survey that they participated in training.*® We use 
these sources to relate program-reported information on the first training after random 
assignment with self-reported information on employment three years after random assignment. 


e Only about 16 percent of treatment group survey respondents first enrolled in long- 
term training. 


Based on administrative records, 84 percent of the sample first enrolled in short-term training. 
The largest share of first training after random assignment was short-term training for Nursing 
Aides, Orderlies, and Attendants (Exhibit 8.2). About 8 percent first enrolled in RN training, and 
about 9 percent first enrolled in LVN training. The average training duration for the long-term 
trainings exceeded 12 months, while the average training duration for the short-term trainings 
was approximately 4 months (not shown).%? 


Although one might have expected lower completion rates for longer-term trainings, we find that 
completion rates are higher for that group. About 81 percent of those who first enrolled in long- 
term training reported any training completion compared to 69 percent of those who first 
enrolled in short-term training (not shown). This observation highlights an important point about 
selection: the groups of treatment group participants that take up short-term versus long-term 
training likely have quite different characteristics. 


In addition to examining trainings for the overall sample, we also compare the distribution of 
training programs for the subgroups defined by baseline enrollment in training and education 
level. Based on a hypothesis of substantial subgroup variation, we undertook this descriptive 
analysis to try to understand what might underlie the program’s overall lack of earnings impacts. 


e Those who were already enrolled or had some postsecondary experience at baseline 
were more likely to enroll in long-term training. 


Exhibits 8.3 and 8.4 report on training type for enrollment and education subgroups. Those who 
could be considered less prepared for HPOG were much more likely to enroll in short-term 
training—more than half of those who were not enrolled in school at baseline or entered with a 
high school education or less at baseline pursued training for Nursing Aides, Orderlies, and 
Attendants. On the other hand, a much larger share of those already enrolled in training or with 
some postsecondary education at baseline pursued long-term training. Less than 10 percent of 
those not enrolled at baseline and those with a high school education or less chose to enroll in 
long-term training. 


38 Of the 6,027 treatment group members who completed the survey and did not skip out of any sections, 4,855 
started a training according to the PRS and 4,124 of those started a training according to the three-year survey. Our 
analysis in the remainder of this chapter focuses on these 4,124 respondents. 


39 Chapter 3 reported that average duration for the treatment group was 7.6 months. The analysis in this chapter uses 
a different treatment sample and only focuses on the duration of first training entered into the PRS. The average 
duration of first training for this subset of the treatment group is 5.5 months. The earlier finding could also have 
included non-HPOG trainings. 
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The discussion to this point has focused on the characteristics of the first HPOG training. Now 
we shift focus to the characteristics of employment as measured by the three-year survey. 


e Three years after random assignment, labor market outcomes were more favorable 
for those who had enrolled in long-term training than for those who had enrolled in 
short-term training. 


Three years after random assignment, 89 percent of those who enrolled in long-term training 
were employed, earning an average of $21 per hour; and 75 percent of those who enrolled in 
short-term training were employed, earning an average of $14.50 per hour. Although this 
analysis does not draw on causal impact estimates, these analyses show that the employment 
occupation was strongly correlated with whether training was short-term or long-term. Those 
who took up short- versus long-term trainings are groups with distinct characteristics. Exhibit 8.5 
shows that more than two thirds of those who enrolled in long-term training and reported 
employment in the three-year survey were working as either Registered Nurses or Licensed 
Practical and Licensed Vocational Nurses. 
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Exhibit 8.2: First HPOG Training by Type and Average Duration 
Training duration 
(months) 


Nursing aides, orderlies, and attendants nn 43% 2.0 
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&¢ 
2s 
Ss Registered nurses 5% 13.5 
20 40 60 


Percentage of respondents 


Notes: Sample includes 4,124 treatment group members with reported training in both the HPOG Performance Reporting System (PRS) and the three-year survey. Occupation is based on the first 
occupational training program reported in the PRS. Training duration is based on the PRS where available; for trainings that extended beyond the range of PRS coverage, training duration is from the 
follow-up survey. Training duration is the average among completers only. Percentages may not sum to 100 due to rounding. 


Sources: PRS and three-year survey. 


Abt Associates HPOG 1.0 Three-Year Impacts Report | pg. 68 


8. PATTERNS OF TRAINING & EMPLOYMENT 


Exhibit 8.3: First HPOG Training by Type and Average Duration for Baseline Enrollment Subgroups 
Training duration 
(months) 
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Notes: Sample includes 4,124 treatment group members with reported training in both the HPOG Performance Reporting System (PRS) and the three-year survey. Occupation is based on the first 
occupational training program reported in the PRS. Training duration is based on the PRS where available; for trainings that extended beyond the range of PRS coverage, training duration is from the 
follow-up survey. Training duration is the average among completers only. Percentages may not sum to 100 due to rounding. 

Sources: PRS and three-year survey. 
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Exhibit 8.4: First HPOG Training by Type and Average Duration for Baseline Education Subgroups 
Training duration 


(months) 
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Notes: Sample includes 4,124 treatment group members with reported training in both the HPOG Performance Reporting System (PRS) and the three-year survey. Occupation is based on the first 
occupational training program reported in the PRS. Training duration is based on the PRS where available; for trainings that extended beyond the range of PRS coverage, training duration is from the 


follow-up survey. Training duration is the average among completers only. Percentages may not sum to 100 due to rounding. 
Sources: PRS and three-year survey. 
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Exhibit 8.5: HPOG Employment Occupation by Training Duration 
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Notes: Sample includes 3,867 treatment group members with reported training in both the HPOG Performance Reporting System (PRS) and the three-year survey, and who reported any employment 
in the three-year survey. Employment occupation is the current or most recent occupation reported in the three-year survey, as coded by the U.S. Census Bureau National Processing Center. Hourly 
wage was the last hourly wage reported in the three-year survey. Other occupation includes both healthcare and non-healthcare occupations. Training program length (“Short-Term” and “Long- 
Term”) refers to the first training reported in the PRS. Percentages may not sum to 100 due to rounding. 

Sources: PRS and three-year survey. 
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8.3 Returning to Training 


As noted in the introduction to this chapter, the HPOG 1.0 FOA encouraged grantees to support 
career pathways or career ladders. This is based on the theory that by combining training and 
employment experiences people can build careers and improve their labor market outcomes. 
This feature of the career pathways framework is key for those who start at the bottom of the 
ladder and need to combine training and work experiences to climb up that ladder into higher- 
wage employment in the healthcare sector. 


Under the career pathways framework, it is not necessary for a first training to meet a high- 
wage standard. Instead, in the career pathways framework, that first short-term training might 
be followed by some employment and then starting another short-term training or a longer-term 
training as a means of advancing in a career. 


This section adds to the evidence on patterns of training and employment by exploring the 
patterns of self-reported transitions from the three-year survey with a particular focus on 
returning for a second training following an initial spell of training-to-employment. *° Such a 
pattern would provide evidence that—at least within three years—some people’s experiences 
reflect the career pathways framework. Furthermore, given the low wages of healthcare jobs 
resulting from short-term training, observing such a pattern is crucial for HPOG to result in high- 
wage healthcare employment. 


The analyses in this section are based on data from the three-year survey. Relative to the PRS, 
the three-year survey has two advantages. First, the survey provides a longer follow-up period 
than the PRS.** Second, it captures training outside of HPOG. A limitation of the survey is that it 
does not capture details of multiple trainings within a particular “spell” of training. As such, if 
individuals enroll in multiple trainings back-to-back (e.g., completing CNA training along with an 
additional short-term training to be eligible for a CNA-plus license), then the survey would record 
only one spell of training. We cannot determine the number of trainings completed or credentials 
earned at the spell level. Similarly, we can only determine whether the training spell ended with 
a completion of training or not, which may not reflect the completion status of other trainings 
within the training spell. 


40 Using HPOG administrative data, prior published work considered the extent to which HPOG participants pursue 
and complete multiple trainings. For instance, 18 months after random assignment, 16 percent of HPOG participants 
who had completed one course enrolled in another, and most of this follow-on training was in short-term courses 
following an initial short-term course (Werner et al. 2018). Loprest and Sick (2018) note that about one quarter of 
participants who completed CNA training completed additional short-term training to obtain a CNA-plus license. 

41 PRS data are generally available through June 2016. Data collection was originally scheduled to end on 
September 30, 2015; however, many grantees received no-cost extensions from ACF and continued entering 
participant data into the PRS through June 2016. The length of PRS follow-up ranges from 52 months for the earliest 
enrollees (those randomized in February 2012) to 18 months for the last enrollees (those randomized in December 


2014). 
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Our analyses classify patterns of training and employment in four mutually exclusive transition 
types: 


Returning to Training (RtT) refers to patterns that contain a completed spell of training, 
followed by a spell of employment, followed by another spell of training.** This training- 
employment-training pattern mirrors the concept underpinning the career pathways 
framework: that returning to training after having worked holds promise for longer-term 
career progress. 


Training to Employment (TtE) refers to patterns that contain a spell of training (whether 
completed or not) followed by a spell of employment, but no subsequent move back to 
training. This training-employment pattern is emblematic of standard job training and does 
not yet reflect the aspect of the career pathways framework that encourages engaging 
trainees in stackable credentials, particularly for those at lower rungs of the career ladder 
(although over a longer follow-up period, we might eventually observe a second training 
spell). 

Always in Training (AiT) refers to patterns where all spells are either training or both, 
indicating the participant was in training for all months. 


Other (Oth) refers to any other types of patterns in the data. These patterns include 
respondents who never move from their first non-training status, those who do not pursue 
training, and those who do not find employment after their training. 


Exhibit 8.6 categorizes the hundreds of patterns collected in the survey data into these four 
transition types. The left portion of the exhibit shows the share of patterns in each transition type 
for HPOG treatment group respondents who first enrolled in short-term training; the right portion 
does so for the subset who first enrolled in long-term training. 


42 In creating these patterns, we counted respondents who are in both status as in either training or employment, as 
appropriate, to complete the pattern. For instance, both-employed-both would be considered RtT (with both taking the 
place of training). But training-both-training would be considered AiT. See Appendix H for the full list of patterns. 
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Exhibit 8.6: Patterns of Training and Employment 
100 
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Notes: Sample includes 4,124 survey respondents in the treatment group who enrolled in training according to the PRS and the survey. 
Percentages may not sum to 100 due to rounding. See Appendix H for a full listing of the patterns that make up each of the four categories. 
Source: Three-year survey. 


e Less than 10 percent of treatment group respondents return to training for a second 
spell after employment, whether their first training was short-term or long-term. 


Over the three-year follow-up, about two thirds of the treatment group respondents had a 
pattern that included a move from training to employment, regardless of whether they first 
enrolled in short-term or long-term training. About 5 percent of the group was in training the 
entire time. 


Across both groups, less than 10 percent of respondents returned for a second spell of training. 
This finding reflects returning for any additional training, which is an upper bound for training 
that we might expect to have a stronger influence on earnings—long-term training. The share of 
respondents who experienced TtE or RtT differs only slightly between those who enrolled in 
short-term versus long-term training. 


8.4 Summary 


The patterns summarized in this chapter describe the experiences of the HPOG treatment 
group through about 36 months after randomization. Our analyses focused on understanding 
how these experiences differed for those enrolled in short-term versus long-term training and to 
what extent these experiences align with the career progression and career ladders defined in 
the HPOG 1.0 FOA (HHS/ACF/OFA 2010). That framework expects trainees to combine 
training resulting in credentials with employment—and thereby to craft a career. Because we 
focus only on the treatment group, the evidence we examine does not permit us to draw 
conclusions about the impact of HPOG on the prevalence of particular patterns. 
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A large portion of the sample completed only short-term training (See Exhibits 8.1 and 8.2). As 
would be common in other, similar samples of individuals coming to training programs, during 
the three-year follow-up for the HPOG treatment group respondents, we observe an increase 
over time in employment; a decrease over time in being in training or being both employed and 
in training; and roughly no change over time in experiencing neither training nor employment. 


At the individual participant level, this non-experimental analysis found that employment 
outcomes three years after random assignment were more favorable for those who first enrolled 
in long-term as opposed to short-term training. The long-term training group consisted of mostly 
those who were enrolled in training at baseline or entered with some postsecondary education 
or more. This is consistent with the subgroup findings reported in Chapter 6. Over the longer 
term, this seeming bifurcation within the HPOG sample might result in different trajectories for 
the two subgroups that will warrant follow-up. 


We also assessed the extent to which HPOG participants returned from post-training 
employment to a second spell of training. This adds to prior literature on the extent to which 
HPOG participants pursued more than one training (Loprest and Sick 2018; Werner et al. 2018). 
Although many HPOG participants transitioned from training to employed status at some point 
over the three-year follow-up, less than 10 percent of them moved from post-training 
employment to a second spell of training. This suggests that—at least within the three-year 
follow-up window—treatment group members are not following the career ladder path described 
by the FOA. Surprisingly, we find a similarly small fraction of the sample returning for a second 
spell of training regardless of whether they enrolled in short-term or long-term trainings. Those 
who completed short-term training have had more time during the follow-up period to pursue 
employment and return for follow-on training than those who completed long-term training (8 
more months on average), but there is no evidence of returning to training at different rates. 
This finding could be driven by a multitude of factors. As noted above, those who took up short- 
versus long-term trainings have distinct characteristics, and the amount of time is only one 
factor in the decision to pursue additional follow-on training 


These findings have implications for assessing the performance of the HPOG logic model and 
the career pathways framework. One perspective is that despite the FOA guidance, HPOG 1.0 
grantees did not urge participants to return for additional training (Werner et al. 2018), nor did 
the large share of participants return for additional training, at least within the three-year follow- 
up window. This may be related to how HPOG 1.0 programs were designed, in accordance with 
grantee performance measures. HPOG 1.0 grantees received “credit” toward their performance 
measures for serving a single person, no matter how many trainings he or she took. This may 
have incentivized grantees to offer more short-term training opportunities with fewer follow-on 
trainings.*? Another possibility is that returning to school is too difficult for this population (even if 
the incentives were structured appropriately). A third (not mutually exclusive) possibility is that 
three years is not long enough for people to engage in a second training. We plan to return to 
this issue and reassess these patterns at the planned six-year follow-up. 


43 HPOG 2.0 revised the incentives to place greater emphasis on follow-on training by allowing grantees to receive 
“credit” for each training completed. 
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The HPOG 1.0 Impact Study uses an experimental evaluation design to provide strong 
evidence of the impacts of 42 local programs operated by 23 HPOG 1.0 grantees nationwide. 
These impacts assess the effectiveness of HPOG in pursuing its dual goals of providing training 
opportunities for low-income individuals and TANF recipients and of providing a skilled 
workforce to meet the needs of the healthcare sector. The large study sample means the study 
can detect relatively small impacts, if they exist. It also permits analysis of impacts on a variety 
of subgroups, as identified at baseline. 


This concluding chapter begins by summarizing the HPOG 1.0 impacts to date. It then 
discusses potential explanations for the impact findings as well as implications for future 
research, policy, and practice. 


9.1 HPOG 1.0 Impact Findings to Date 


This section summarizes HPOG’s impacts to date across the main outcome domains at the 
short-term and three-year follow-ups. The comparisons between the short-term and three-year 
impact results discussed here are purely qualitative; we have not conducted tests of whether the 
two set of values being compared are statistically different from each other. As a result, the 
differences between two values being compared may or may not represent statistically 
significant differences in impact. We discuss these results for the sample overall and for major 
subgroups of interest, summarizing them in Exhibit 9.1. 


9.1.1 Confirmatory Outcomes 


Educational Progress 


As of the short-term follow-up (about 15-18 months after random assignment), the HPOG 1.0 
Impact Study reported that the treatment group’s educational progress, defined at that point as 
having completed or still being enrolled in training, had increased 7 percentage points, relative 
to what would have occurred in the absence of the HPOG Program. As of the three-year follow- 
up, this impact was 13 percentage points, on the narrower educational progress measure of 
training completion.** The study’s Short-Term Analysis Plan (Harvill, Moulton, and Peck 2015) 
and Three-Year Analysis Plan (Litwok et al. 2018) pre-specified each of these educational 
progress outcomes as “confirmatory,” meaning that, for each respective report, these outcomes 
are the main indicators of the extent to which the program is making progress toward its goals. 


At the short-term follow-up, HPOG had larger impacts on educational progress for those who 
were receiving TANF or SNAP/WIC at baseline, relative to those who were not receiving any 
public assistance at baseline. As of the three-year follow-up, HPOG had a greater number of 
differential impacts on educational progress, with all of them favoring the subgroups who could 
be considered less prepared at baseline for a program such as HPOG, such as those who were 
not enrolled in training or had lower levels of education. 


44 For a direct comparison, the Short-Term Impacts Report included training completion as an exploratory outcome 
and reports a 9 percentage point impact on that measure (58 percent completion in the treatment group versus 49 
percent completion in the control group). 
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Earnings 

The evaluation team also pre-specified and registered earnings as a secondary outcome in the 
short-term and as a confirmatory outcome for the three-year follow-up study. The short-term 
report detected a small earnings impact in the fifth follow-up quarter. However, there was no 
detectable impact of HPOG on earnings in the 12th and 13th quarters after random assignment. 


In the short-term, those with no barriers to school or work had larger earnings impacts, relative 
to those who had some barriers. At the three-year follow-up point, those who came into the 
program with some postsecondary education had larger earnings impacts, relative to those with 
less education. These two subgroup findings suggest that those who are seemingly better 
prepared to take advantage of HPOG’s offerings (because of their baseline barriers or 
education) experience larger earnings impacts. 


Exhibit 9.1: Summary of HPOG 1.0’s Short-Term and Three-Year Impacts, Overall and by 
Subgroup 


Educational Quarterly Employment in Job Public 
Outcome Domain: Progress@ Earnings Employment Healthcare Quality Assistance 
(ppt) ($) (ppt) (ppt) (ppt) (ppt) 
Analysis Timing: Short 3-Year Short 3-Year Short 3-Year Short 3-Year Short 3-Year Short 3-Year 
Overall Impacts 
“7 M13 $137 “1 Mil 12 M2 
ubgroup Impacts 
Education: 
SHS vs. postsecondary €5 >$303 >5» <4 
Enrolled: 
Not enrolled vs. enrolled <7 €5 <6 33 36 
Employed: 
Not employed vs. employed <6 <4 €5 
Barriers: 
Some barriers vs. none €5 >$251¢ >3 
Public Assistance: 
TANF vs. SNAP/WIC <6 
TANF vs. no assistance <7 €11 
SNAP/WIC vs. no assistance <4 <7 €8 
Dependent children: 
Children vs. no children €3 


Notes: “ppt” refers to percentage point. Overall impact arrows indicate the direction of the impact. Subgroup impact arrows indicate which of the 
two subgroups (left vs. right) had the larger impact, by the amount indicated. 

aThe short-term confirmatory measure of educational progress was training completion or ongoing enrollment, and the three-year confirmatory 
measure of educational progress is training completion. 

> The main subgroup analysis in the short-term report included four education-defined subgroups. Job quality impacts were 5 percentage points 
larger for those with some postsecondary versus a high school diploma and 6 points larger for those with a college degree versus those with a 
high school diploma. 

¢ The main subgroup analysis in the short-term report included three barriers-defined subgroups. The impact was $425 larger for those with no 
barriers at baseline relative to those with two or more barriers. 


9.1.2 Secondary and Exploratory Outcomes 


Beyond these confirmatory outcomes, the evaluation also considered several secondary and 
exploratory outcomes. Here we discuss two: employment and receipt of public assistance. 
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Employment 


At both the short-term and three-year follow-up points, employment rates were similar between the 
treatment and control groups. At the fifth follow-up quarter, about 70 percent of the study sample 
was employed. At the 12th-13th follow-up quarters, more than 80 percent were, with the HPOG 
group employed at a rate 1 percentage point greater than the control group. 


HPOG’s impacts on employment in healthcare were much larger, signifying a shift in the type of jobs 
the treatment group secured. As of the short-term follow-up, 53 percent of the treatment group, 
about 11 percentage points more than the control group, was employed in healthcare. By the three- 
year follow-up, about 56 percent of the treatment group was employed in healthcare, 12 percentage 
points more than in the control group. 


Exhibit 9.1 also indicates employment impacts for subgroups. In the short-term, the impact for 
healthcare employment was larger for those who at baseline were not enrolled in training or not 
employed, relative to their peers who at baseline were enrolled in school or employed. In 
comparison, after three years, those with some barriers to school or work experienced larger total 
employment impacts relative to those with no barriers; we observe no differential subgroup impacts 
on healthcare employment at three years. These subgroup findings are not obviously consistent 
with one another or necessarily with prior expectations, and so we have interpreted them tentatively. 


Public Assistance 


In both the short-term and after three years, HPOG did not reduce public assistance use overall, 
although we do observe some differential impacts. In the short-term, those study participants who 
entered a local HPOG program already enrolled in school experienced a 3 percentage point larger 
increase in their public assistance use than did those not yet enrolled. After three years, in addition 
to those enrolled in school at baseline, those with less education at baseline as well as those 
receiving TANF at baseline (relative to SNAP/WIC) also experienced relatively larger increases in 
their public assistance use. 


9.2 Discussion of Impact Findings 


Local HPOG programs operated in a service right environment with a highly motivated control 
group, which implies that the HPOG Program had to surpass a relatively high bar in order to 
generate impacts. In that context, HROG generated improvements in educational progress, though 
these improvements did not result in earnings impacts after three years. On the one hand, these 
results might seem surprising, considering that the conventional logic model implies that training 
leads to higher earnings. On the other hand, these results might not seem surprising, given the 
heterogeneity within the sample—in terms of their experiences with training duration and in the labor 
market for the treatment group, and in terms of demographic and other characteristics for the 
sample overall. 


We discuss these findings using four potential explanations for why educational progress gains 
have not yet and may or may not in the future result in earnings gains: 


1. Training duration seems to matter. A large majority of the treatment group enrolled in short- 
term trainings. Three years after random assignment, those who had initially enrolled in short- 
term trainings had lower wages than those who had initially enrolled in longer-term trainings, 
according to descriptive analyses. 
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2. Subgroups seem to matter. Those who were seemingly less prepared at baseline for the 
longer-term trainings available in a program such as HPOG made relatively more educational 
progress than those who began training more prepared. However, those who were seemingly 
more prepared at baseline had larger labor market impacts. 


3. Implementation seems to matter. The specific execution of HPOG 1.0 grants in practice— 
including the grant’s timing and performance metrics—may, at least in part, explain findings 
about training duration and the extent of follow-on trainings. 


4. More time is needed. It could be the case that three years after randomization is not enough 
time for impacts on labor market outcomes to materialize. 


We elaborate on each of these as possible, partial explanations for this report’s findings. 


9.2.1 Training Duration Seems to Matter 


The FOA encouraged local HPOG programs to “provide eligible individuals with the opportunity to 
obtain education and training for occupations in the healthcare field that pay well and are expected 
to either experience labor shortages or be in high demand” (HHS/ACF/OFA 2010, p. 1). As they 
implemented HPOG, grantees offered a wide variety of trainings—some of them leading to 
healthcare sector jobs with higher wages (e.g., Registered Nurse, Licensed Practical Nurse, 
Licensed Vocational Nurse) and some of them leading to entry-level healthcare sector jobs (e.g., 
Nursing Aides, Orderlies, Attendants) with lower wages (Chapter 8).*° Although the offerings were 
expansive, the overwhelming share of participants enrolled in training for entry-level healthcare jobs. 


As reported in Chapter 8, among those who enrolled in short-term training (about 84 percent of 
treatment group participants), 75 percent were employed at the three-year follow-up and they 
earned an average wage of $14.50 per hour. In comparison, among those who enrolled in long-term 
trainings (about 16 percent of treatment group participants), 89 percent were employed at the three- 
year follow-up and they earned an average of $21 per hour. This distribution—of sample, trainings, 
and associated jobs—likely contributes to the evaluation’s summary finding of no detectable 
earnings impacts despite meaningful educational progress impacts. 


Because the analysis on which this explanation is based is descriptive, future analysis would be 
needed to tease out the implications of this heterogeneity, including identifying the impacts on 
wages for those who took up short-term versus long-term trainings. 


9.2.2 Subgroups Seem to Matter 


Our subgroup analyses show that training duration and subsequent wages are related to baseline 
characteristics. Educational progress gains surface in all subgroups—and in some cases are even 
greater for those who are seemingly less prepared for a program such as HPOG. However, the 
labor market gains accrue only to those who had traits at baseline commonly associated with labor 
market success, such as higher levels of education or fewer barriers to school or work. Reinforcing 
the training duration argument in the previous section, the seemingly better prepared group was 
more likely to engage in long-term training. The findings for the better prepared group are consistent 


45 In an analysis of supply and demand for Certified Nursing Assistant (CNA) jobs, Loprest and Sick (2018) note that 
CNA training is a good fit for relatively disadvantaged groups because the academic requirements are light and 
training is short. Although demand for CNAs is high, CNAs earn low wages. 
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with expectations for career pathways training—that participants’ educational progress results in 
well-paying jobs. 


In comparison, the findings for the seemingly less prepared group imply that training completion— 
particularly for short-term trainings—may not be enough to translate to impacts on earnings for that 
group. Such individuals likely begin on the lowest rungs of the career ladder. Starting from that 
position, to complete the follow-on training needed for a well-paying healthcare job might be 
extremely challenging, given difficult prerequisites and other barriers to training such as its cost or 
family responsibilities (Loprest and Sick 2018). 


9.2.3. Implementation Seems to Matter 


Another explanation for the combination of findings pertains to the specifics of how HPOG 1.0 
grantees executed the grants in practice. This was a five-year grant program, where awards were 
not expected to be renewed beyond five years. According to HROG Program guidance, grantee 
performance was measured by counting individuals, not trainings completed. Furthermore, 
individuals in our sample only began enrolling in training midway through the grant period, leaving at 
most two and a half years during which the earliest enrollees could access the training that was 
HPOG grant-funded. If programs encouraged people to get work experience before coming back for 
more training (and some grantees did encourage that), then the time-limited nature of the grant 
would make gaining additional training difficult to accomplish. Certainly grantees did not want to 
have people in the middle of training when their funding ended, and the evaluation observed 
grantees urging participants toward shorter-term trainings especially toward the end of the grant 
period (e.g., Werner et al. 2018). 


Together, these issues in the way that HPOG 1.0 operated might have created an incentive for 
short-term trainings and not necessarily for follow-on trainings. Given the guidance in the FOA—to 
generally “support career pathways” or “articulated career ladders” without an explicit 
requirement for long-term or follow-on trainings—this implementation might have been expected. 
These are issues that the HPOG 2.0 FOA addressed, being more explicit about HPOG 2.0 
grantees’ offering longer-term and follow-on trainings (HHS/ACF/OFA 2015). 


These implementation factors provide a possible explanation for the observations that a greater 
share of trainings completed were short term, and only a small share of participants participated in 
follow-on trainings. These observations subsequently provide a possible explanation for the impact 
results of meaningful educational progress that has not yet translated into earnings gains. 


9.2.4 More Time Is Needed 


Finally, perhaps it takes more than three years for educational progress to yield meaningfully higher 
earnings. Indeed, all of the trends discussed in the prior three sub-sections warrant examination at 
longer-term follow-up. Recent evidence from Project QUEST reveals mid-term (three-year) null 
results, with longer-term (beginning after four years) favorable impacts on earnings (Roder and Elliot 
2019). Other evaluations find mixed evidence on labor market impacts in the mid-term (e.g., Fortson 
et al. 2017; Schwartz, Strawn, and Sarna 2018), where longer-term results are still forthcoming. 


For HPOG 1.0, earnings impacts might arise in the longer term if a substantial share of the sample 
moves into better healthcare jobs, an expectation based on the assumption that healthcare sector 
jobs can be better paying than whatever jobs control group members would hold. This might happen 
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simply through the passage of time, or it might happen if a greater share of treatment group 
members return to participate in additional training. That additional training would not be in HPROG 
1.0 programs (that funding has ended), but longer-term earnings gains are plausible if HPOG 1.0 
programs have instilled in their participants that a life’s career is built through returning to education 
and training as a means for advancement.*® 


Without movement into better healthcare employment, either through promotion or via additional 
training, we would not expect impacts on longer-term earnings to change from those in this report. In 
fact, if the share of the treatment group that is in entry-level, lower-paying healthcare jobs remains 
relatively stable while more of their control group counterparts advance (whether in healthcare or 
non-healthcare jobs), then negative impacts on earnings might well emerge in the longer term. 


9.3. Implications for Future Research, Policy, and Practice 


This evaluation reveals that the collection of local HPOG programs in the HPOG 1.0 Impact Study 
improved educational progress outcomes for the entire study sample and had no detectable impact 
on earnings as of the three-year follow-up. Compared to other studies of job training programs, this 
evaluation has a very large sample; consequently, the failure to detect impacts on earnings is not 
due to a simple lack of statistical power. 


The collection of HPOG 1.0 programs is quite diverse, with their commonality being the stream of 
funding that supported them. As such, the HPOG Impact Study can be thought of as an 
“effectiveness” evaluation, referring to its test of a program under “real-world” conditions. In 
comparison, most other evaluations of similar sector- and/or career pathways-based training 
programs are of programs that were selected as a high quality version of the program. As 
interventions being tested “under ideal and controlled circumstances,” these can be thought of as 
“efficacy” evaluations (e.g., Singal, Higgins, and Waljee 2014). We mention this effectiveness- 
efficacy distinction because it is relevant to interpreting the impacts from the two types of studies. 
Evaluations of selected programs (efficacy trials) are generally expected to detect greater impacts 
than would evaluations of large funding streams of programs operating under real-world conditions. 
The nature of the HPOG impact evaluation (as an effectiveness evaluation) leads to the conclusion 
that an entire funding stream—uwith all of the heterogeneity of programs included—improved the 
educational progress of its participants. 


That said, that HPOG’s impacts on educational progress did not translate into impacts on earnings 
after three years leaves open the question of whether the additional training afforded the treatment 
group will ever translate into significant earnings gains. The previous section considered some 
plausible explanations for this combination of findings. Whether earnings gains would materialize in 
the future seems dependent on whether those HPOG participants currently in low-paying healthcare 
sector jobs return to complete additional training at a rate higher than what the control group 
accomplishes in the same time period. Doing so might help them advance along a healthcare 
career ladder, where the higher rungs do indeed have higher pay. 


The planned six-year follow-up will reveal which of these possibilities becomes reality for the HPOG 
sample. If the treatment group does not return to training and remains in the largely low-paying 


46 When the HPOG 1.0 grants ended, some programs continued via funding from other sources, including HPOG 2.0. 
HPOG 1.0 participants might have pursued additional training either at the same location where they received their 
HPOG 1.0 training, if it was still in operation, or from some other provider. 


Abt Associates HPOG 1.0 Three-Year Impacts Report | pg. 81 


9. DISCUSSION & CONCLUSION 


healthcare jobs they are in as of the three-year follow-up, then future earnings gains seem unlikely. 
It is possible that the increase in lower-rung training may provide participants an advantage in 
employment and earnings in a less robust economy. Evaluation results are never timeless or 
placeless, and so additional research may provide insights on how these programs function across 
varied iterations of them across time and place. 


A report based on those long-term evaluation efforts is scheduled for release in 2021 and can follow 
up on the issues raised in this three-year analysis but with longer-term data. In addition to longer- 
term HPOG results, ACF is funding intermediate and longer term follow-up of all nine programs 
included in the PACE Study. Those results will be released on a rolling basis through 2021. 
Additionally, ACF has funded a comprehensive evaluation of the second round of HPOG grants 
(HPOG 2.0). That study also includes a cost-benefit analysis, which will have value in helping to 
understand—given the impacts observed—the extent to which the government investment is 
worthwhile. Together, this rich body of research will add substantially to the field’s collective 
knowledge about the impacts of career pathways programs, including various implementations of 
them. 


The analysis of the three-year findings offers some implications for policy and practice, as well. For 
example, because this analysis found 


e that training durations were generally short (a large majority of the sample took up training in 
short-term programs, averaging four months to completion), 


e ameaningful shift in healthcare employment that came with no increase in earnings, and 


e very low rates of returning to training, 
future HPOG and similar programs may want to 


e emphasize and support longer-term training for mid- to higher-level jobs in healthcare, and 


e encourage and support program graduates of short-term programs who are in low-paying jobs 
to take advantage of additional trainings, which may provide opportunities for them to move up 
the career ladder. 


Indeed, the HPOG 2.0 FOA emphasized longer-term and follow-on trainings to a greater degree, 
implying that its evaluation will be poised to offer evidence on the effectiveness of that more fully 
articulated career pathways model. 


Nevertheless, even in relatively service-rich environments and with a well-motivated and high 
achieving control group, HPOG increased training completion and provided financial assistance and 
support services that led to some better outcomes for the treatment group as a whole, as well as for 
some major subgroups. Although a lesser share of the HPOG treatment group reports experiencing 
material hardship, and a greater share are in healthcare sector employment (which has the potential 
to improve their earnings), these impacts have not translated into meaningful labor market gains on 
average as of the three-year follow-up. Because a primary motivation for individuals getting job 
training—and for government programs supporting job training—is that more training leads to higher 
earnings, earnings impacts will be an important marker of success to observe over time. 
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